Une place pour une véritable innovation. Partagez vos propres utilitaires créés avec la communauté Manjaro.
Questions et discussions sur la programmation et le codage.
Répondre

pacman-mirrors une alternative simpliste...

#1Messageil y a 3 semaines

:bjr:

Ayant du mal à comprendre le fonctionnement et surtout comment gérer les options de pacman-mirrors (il faut un doctorat en informatique !), je me suis donc amusé à écrire une alternative beaucoup plus simple
  • Il est basé sur la prochaine version de pacman-mirrors avec des appels asynchrones - Ce qui le rend extrêmement rapide.
    Si rapide qu'il n'y a plus lieu de sélectionner un pays : donc on ne sélectionne (automatiquement) qu'un continent
  • Il utilise geoIP pour trouver le pays et continent du demandeur, mais il est possible de sélectionner un continent manuellement
  • Il sélectionne uniquement les miroirs à jour de notre continent
  • Il sélectionne les miroirs de notre branche (pas testé car je n'ai que stable !!!)
  • Il met en tête de liste les miroirs de notre pays (à cause du timeur...)
  • Il lance les tests en parallèle et se stoppe au bout de 5 secondes (par défaut)
  • Il sauvegarde uniquement si il est lancé avec "sudo", il fait une sauvegarde avant
Une sortie avec un lancement sans aucun paramètre :

90 mirrors (all)
52 mirrors in continent: Europe
24 mirrors in branch: stable
sort by continent (first:Europe) and country (first:France)
...
Resume all
0.18645739555358887      http://kibo.remi.lu/
0.26831841468811035      http://mirror.catn.com/pub/manjaro/
0.32701778411865234      http://manjaro.mirrors.uk2.net/
0.34636378288269043      http://ftp.free.org/mirrors/repo.manjaro.org/repos/
0.39118218421936035      http://ftp.snt.utwente.nl/pub/linux/manjaro/
0.3985757827758789       http://mirror.easyname.at/manjaro/
0.4525022506713867       http://mirror.datacenter.by/pub/mirrors/manjaro/
0.46449899673461914      http://mirrors.colocall.net/manjaro/
0.4885101318359375       http://ftp.vectranet.pl/manjaro/
0.5062437057495117       http://mirror.futureweb.be/manjaro/
0.5090365409851074       http://mirror.terrahost.no/linux/manjaro/
0.5160796642303467       http://ftp.tu-chemnitz.de/pub/linux/manjaro/
0.6179747581481934       https://mirror.neostrada.nl/manjaro/
0.6668210029602051       https://manjaro.moson.eu/
0.837090015411377        http://mirror.truenetwork.ru/manjaro/
0.9191088676452637       http://mirror.inode.at/manjaro/
1.0456562042236328       https://quantum-mirror.hu/mirrors/pub/manjaro/
1.12028169631958         https://nova.quantum-mirror.hu/mirrors/pub/manjaro/
1.1471326351165771       https://ftp.caliu.cat/pub/distribucions/manjaro/
1.2209060192108154       https://mirror.23media.com/manjaro/
1.6292178630828857       http://mirror.ragenetwork.de/manjaro/
2.285478353500366        https://manjaro.grena.ge/
--             http://ftp.linux.org.tr/manjaro/
--             http://repo.manjaro.org.uk/
::Total duration: 4.007263660430908
Saved in /etc/pacman.d/mirrorlist, backup in /etc/pacman.d/mirrorlist~
les "--", c'est juste que le timeur de 4 secondes a interrompu les tests - pas grave on a déjà largement les plus rapides (22 miroirs en 4 secondes sur une ligne adsl)


Pour "jouer", il est possible de passer quelques paramètre :

./mirrors-update.py AS
EU,AS,AF,OC,NA,SA (AFrique, Nord Amérique, OCéanie...) ici on force un autre continent que celui donné automatiquement par notre ip

Si le timeout de 5 secondes est trop petit :maisnon: , il est possible de lui en passer un différent :

./mirrors-update 16
Il est possible d'avoir un fichier ./logs.txt, histoire d'avoir un historique et surtout voir si on a (presque) toujours les mêmes résultats :

./mirrors-update --log
Et pour les (gros) joueurs, il est possible de ne pas se limiter à un seul continent - option bien sûr sans aucun intérêt pour nous :

./mirrors-update -a
ATTENTION:
Il ne teste que l'url et pas la validité du protocole (https,http,ftp)
Il ne modifie pas la branche

---

#!/usr/bin/python
"""
mirrors-update.py
v 1.0.1 - 27/10/2019
"""
import sys
import os
from urllib import request
import json
import time
import datetime
import locale
from enum import IntEnum

MAXDOWNLOAD = 50000 # read only 50 000 octets
max_workers = 8

continents = {
    "AF": "Africa",
    "AN": "Antarctica",
    "AS": "Asia",
    "EU": "Europe",
    "NA": "North America",
    "OC": "Oceania",
    "SA": "South America"
}

countries = {
    'Australia':'OC',
    'Bangladesh':'AS',
    'Brazil':'SA',
    'Canada':'NA',
    'Chile':'SA',
    'China':'AS',
    'Costa_Rica':'SA',
    'Colombia':'SA',
    'Ecuador':'SA',
    'Hong_Kong':'AS',
    'India':'AS',
    'Indonesia':'AS',
    'Iran':'AS',
    'Japan':'AS',
    'Kenya':'AF',
    'New_Zealand':'OC',
    'Philippines':'AS',
    'Singapore':'AS',
    'South_Africa':'AF',
    'South_Korea':'AS',
    'Taiwan':'AS',
    'Thailand':'AS',
    'United_States':'NA',
    'Vietnam':'AS',
}

class Branches(IntEnum):
    stable = 0
    testing = 1
    unstable = 2

config = {
    'country': 'France',
    'continent_code': 'EU',
    'continent': 'Europe',
    'max_workers': 8,
    'debug' : False,
    'log': False,
    'timeout' : 5,          # stop all tests after X seconds
    'branch': Branches.stable,
    'count': 3,              # want only 3 first speed
}

def usage():
    print("Usage: ./mirrors-update.py [continent code] [-a] [timeout]")
    print("-a: \t\t all mirrors")
    print("continent code: \t filter (EU,AS,AF,OC,NA,SA)")
    print("timeout: for stop tests (default 4 seconds)")
    print("")
    print("ENV: \"count\": save x mirrors (3 by default)")
    exit(0)

def get_parameters(config):
    if '-h' in sys.argv or '--help' in sys.argv:
        usage()
    if '--debug' in sys.argv:
        config['debug'] = True
        config['log'] = True
    if '--log' in sys.argv:
        config['log'] = True
    config['count'] = int(os.getenv('count', 3))
    config['country'], config['continent_code'] = get_my_position()
    for arg in sys.argv:
        if arg in continents.keys():
            config['continent_code'] = arg
            config['continent'] = continents.get(arg)
    for arg in sys.argv:
        if arg.isdigit():
            config['timeout'] = int(arg)

    with open("/etc/pacman-mirrors.conf") as conf_file:
        for line in conf_file:
            if line.startswith("Branch"):
                (_, value) = line.split("=", 1)
                value = value.strip()
                try:
                    config['branch'] = Branches(Branches._member_names_.index(value))
                except ValueError:
                    pass
    return config

def get_my_position(ip='') ->str:
    """ param ip : only for tests """
    if ip:
        ip = f"/{ip}"
    try:
        with request.urlopen(f"https://get.geojs.io/v1/ip/geo{ip}.json") as f_url:
            req = json.loads(f_url.read())
            return req['country'], req['continent_code']
    except Exception:
        return '', ''
    return '', ''

def filter_continent(mirrors: list, continent: str) ->list:
    if not continent:
        return mirrors
    # or can use ?  pacman_mirrors/constants/timezones.py
    return [m for m in mirrors if countries.get(m['country'], 'EU') == continent]


def load_datas(url: str) -> list:
    """ load url list """
    with request.urlopen(url) as f_url:
        req = f_url.read()
    mirrors = json.loads(req)
    print(f"{len(mirrors)} mirrors (all)")
    for mirror in mirrors:
        mirror['timer'] = 99    # by default all bad
        mirror['continent_code'] = countries.get(mirror['country'], 'EU')
    return mirrors

def download_data(mirror: dict)->int:
    """ NOT use protocols !"""
    url = f"{mirror['url']}stable/core/x86_64/core.db"
    start = time.time()
    with request.urlopen(url, timeout=3) as f_url:
        _ = f_url.read(MAXDOWNLOAD)
    end = time.time()-start
    mirror['timer'] = end
    print(f"  => {end:.3f}\t{mirror['url']}")
    return end

def test_threads(timeout: int) ->int:
    from concurrent.futures import ThreadPoolExecutor
    import concurrent.futures
    print("::Speed tests...")
    startthread = time.time()

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        future_to_url = {executor.submit(download_data, url): url for url in mirrors}
        try:
            for _ in concurrent.futures.as_completed(future_to_url, timeout=timeout):
                pass
        except KeyboardInterrupt:
            executor._threads.clear()
            concurrent.futures.thread._threads_queues.clear()
            return time.time()-startthread
        except concurrent.futures._base.TimeoutError as err:
            print(f"::Timeout: {err}", file=sys.stderr)
            executor._threads.clear()
            concurrent.futures.thread._threads_queues.clear()

    end = time.time()-startthread
    print(f"::Total duration: {end}")
    return end

def save_log(mirrors: list, url: str, title: str):
    """ save in log for view if we have always the same first url ..."""
    mirrors.sort(key=lambda mirror: mirror['timer'], reverse=False)
    with open(url, 'a') as flog:
        flog.write(f"{title}\t")
        for i, mirror in enumerate(mirrors):
            flog.write(f"{mirror['url']}\t")
            if i > 2:
                break
        flog.write("\n")

def save_mirrorlist(filename: str, mirrors: list, branch: str, maxi=12)->bool:
    """ save bests in etc """
    from shutil import copyfile
    if not mirrors:
        return False
    if maxi < 1:
        maxi = 1

    try:
        copyfile(filename, f"{filename}~")
    except Exception as err:
        print("ERROR:", err, file=sys.stderr)
        sys.exit(2)
    try:
        with open(filename, "w") as outfile:
            outfile.write(f"##\n# Manjaro Linux default mirrorlist\n# {datetime.date.today().strftime('%x')}\n##\n\n")
            for i, mirror in enumerate(mirrors):
                if i >= maxi:
                    continue
                country = mirror['country']
                if country.startswith('!'):
                    country = country[1:]
                outfile.write(f"## Country : {country}\n")
                outfile.write(f"Server = {mirror['url']}{branch}/$repo/$arch\n\n")

    except OSError as err:
        print("ERROR:", err, file=sys.stderr)
        sys.exit(2)
    print(f"\n::Saved in {filename}, backup in {filename}~")
    return True


if __name__ == '__main__':
    locale.setlocale(locale.LC_ALL, '')

    config = get_parameters(config)
    print("parameters:", config)

    mirrors = load_datas("https://repo.manjaro.org/status.json")
    #print(mirrors)

    if '-a' not in sys.argv:
        mirrors = filter_continent(mirrors, config['continent_code'])
        print(f"{len(mirrors)} mirrors in continent: {config['continent']}")

    mirrors = [m for m in mirrors if m['branches'][int(config['branch'])] == 1 and 'http' in m['protocols']]
    print(f"{len(mirrors)} mirrors in branch: {config['branch'].name}")

    print(f"sort by continent (first:{config['continent']}) and country (first:{config['country']})")
    for mirror in mirrors:
        if mirror['country'] == config['country']:
            mirror['country'] = f"!{mirror['country']}"
        if mirror['continent_code'] == config['continent_code']:
            mirror['continent_code'] = f"!{mirror['continent_code']}"
    mirrors.sort(key=lambda mirror: mirror['country'], reverse=False)
    mirrors.sort(key=lambda mirror: mirror['continent_code'], reverse=False)
    #print(mirrors)

    duration = test_threads(config['timeout'])

    # sort mirrors by timers """
    mirrors.sort(key=lambda mirror: mirror['timer'], reverse=False)
    print("\n::Resume all")
    for mirror in mirrors:
        if mirror['timer'] > 90:
            print(f"--   \t{mirror['url']}", file=sys.stderr)
        else:
            print(f"{mirror['timer']:.3f}\t{mirror['url']}")
    print(f"::Total duration: {duration}")
    mirrors = [m for m in mirrors if int(m['timer']) < 30]

    if config['log']:
        save_log(mirrors, "./logs.txt", duration)

    if os.geteuid() == 0 and mirrors:
        # if root save in /etc/pacman.d/mirrorlist
        save_mirrorlist("/etc/pacman.d/mirrorlist", mirrors, config['branch'].name, config['count'])

pacman-mirrors une alternative simpliste...

#2Messageil y a 3 semaines

Intéressant :bjr:

]$ ./pacman-mirrors-papajoke 
parameters: {'country': 'France', 'continent_code': 'EU', 'continent': 'Europe', 'max_workers': 8, 'debug': True, 'log': True, 'timeout': 4, 'branch': <branches.testing: 1>, 'count': 3}
90 mirrors (all)
52 mirrors in continent: Europe
28 mirrors in branch: testing
sort by continent (first:Europe) and country (first:France)
::Speed tests
	=> 1.5196287631988525 	 http://manjaro.telecoms.bg/
	=> 1.6344337463378906 	 http://ftp.free.org/mirrors/repo.manjaro.org/repos/
	=> 1.6319527626037598 	 http://mirror.futureweb.be/manjaro/
	=> 1.820539951324463 	 http://mirror.easyname.at/manjaro/
	=> 1.8718080520629883 	 http://kibo.remi.lu/
	=> 1.300225019454956 	 http://ftp.snt.utwente.nl/pub/linux/manjaro/
	=> 2.1747934818267822 	 http://mirror.terrahost.no/linux/manjaro/
Timeout: 21 (of 28) futures unfinished
::Total duration: 4.114170789718628

Resume all
1.300225019454956 	 http://ftp.snt.utwente.nl/pub/linux/manjaro/
1.5196287631988525 	 http://manjaro.telecoms.bg/
1.6319527626037598 	 http://mirror.futureweb.be/manjaro/
1.6344337463378906 	 http://ftp.free.org/mirrors/repo.manjaro.org/repos/
1.820539951324463 	 http://mirror.easyname.at/manjaro/
1.8718080520629883 	 http://kibo.remi.lu/
2.1747934818267822 	 http://mirror.terrahost.no/linux/manjaro/
** error  99 		 http://mirror.inode.at/manjaro/
** error  99 		 http://mirror.datacenter.by/pub/mirrors/manjaro/
** error  99 		 https://manjaro.grena.ge/
** error  99 		 https://mirror.23media.com/manjaro/
** error  99 		 http://mirror.ragenetwork.de/manjaro/
** error  99 		 http://ftp.tu-chemnitz.de/pub/linux/manjaro/
** error  99 		 https://manjaro.moson.eu/
** error  99 		 https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
** error  99 		 https://quantum-mirror.hu/mirrors/pub/manjaro/
** error  99 		 https://nova.quantum-mirror.hu/mirrors/pub/manjaro/
** error  99 		 https://mirrors.opensource.is/manjaro/
** error  99 		 https://mirror.neostrada.nl/manjaro/
** error  99 		 http://ftp.vectranet.pl/manjaro/
** error  99 		 http://ftp.dei.uc.pt/pub/linux/manjaro/
** error  99 		 http://mirror.truenetwork.ru/manjaro/
** error  99 		 https://ftp.caliu.cat/pub/distribucions/manjaro/
** error  99 		 http://ftp.linux.org.tr/manjaro/
** error  99 		 http://mirrors.colocall.net/manjaro/
** error  99 		 http://mirror.catn.com/pub/manjaro/
** error  99 		 http://repo.manjaro.org.uk/
** error  99 		 http://manjaro.mirrors.uk2.net/
::Total duration: 4.114170789718628

pacman-mirrors une alternative simpliste...

#3Messageil y a 3 semaines

:rigole: Avec ta ligne de brousse, 5 secondes, c'est un peu juste :wink:
Pour ne pas passer le timeout en paramètre il est possible de modifier le code source :
ligne 67, on a 5 secondes par défaut

        'timeout' : 5,
En tout cas, avec le tri des miroirs : ton pays détecté en premier, tu t'en sors tout juste (à la micro-seconde :roll: )

ps: cool, ça marche avec autre chose que "stable"

pacman-mirrors une alternative simpliste...

#4Messageil y a 3 semaines

:bjr: Super rapide, beau boulot papajoke :bien:clap

./papjoke-pacman-mirrors
parameters: {'country': 'France', 'continent_code': 'EU', 'continent': 'Europe', 'max_workers': 8, 'debug': True, 'log': True, 'timeout': 4, 'branch': <branches.stable: 0>, 'count': 3}
90 mirrors (all)
52 mirrors in continent: Europe
25 mirrors in branch: stable
sort by continent (first:Europe) and country (first:France)
::Speed tests
	=> 0.16053271293640137 	 http://ftp.free.org/mirrors/repo.manjaro.org/repos/
	=> 0.3029358386993408 	 http://mirror.futureweb.be/manjaro/
	=> 0.37912607192993164 	 http://mirror.inode.at/manjaro/
	=> 0.36032915115356445 	 http://mirror.ragenetwork.de/manjaro/
	=> 0.43140339851379395 	 http://mirror.easyname.at/manjaro/
	=> 0.3841896057128906 	 http://ftp.snt.utwente.nl/pub/linux/manjaro/
	=> 0.47005653381347656 	 http://ftp.tu-chemnitz.de/pub/linux/manjaro/
	=> 0.5718839168548584 	 https://mirror.23media.com/manjaro/
	=> 0.48391079902648926 	 http://mirror.terrahost.no/linux/manjaro/
	=> 0.6305742263793945 	 http://mirror.datacenter.by/pub/mirrors/manjaro/
	=> 0.649099588394165 	 https://mirror.neostrada.nl/manjaro/
	=> 0.713778018951416 	 https://manjaro.moson.eu/
	=> 0.6331231594085693 	 http://ftp.vectranet.pl/manjaro/
	=> 0.94211745262146 	 https://quantum-mirror.hu/mirrors/pub/manjaro/
	=> 0.41837120056152344 	 http://manjaro.mirrors.uk2.net/
	=> 0.5288162231445312 	 http://mirror.catn.com/pub/manjaro/
	=> 1.0911262035369873 	 https://manjaro.grena.ge/
	=> 0.8285701274871826 	 http://ftp.dei.uc.pt/pub/linux/manjaro/
	=> 0.6426975727081299 	 http://mirrors.colocall.net/manjaro/
	=> 1.128014326095581 	 https://nova.quantum-mirror.hu/mirrors/pub/manjaro/
	=> 0.8253636360168457 	 http://ftp.linux.org.tr/manjaro/
	=> 0.918102502822876 	 http://mirror.truenetwork.ru/manjaro/
	=> 0.9560236930847168 	 https://ftp.caliu.cat/pub/distribucions/manjaro/
	=> 1.4240171909332275 	 http://kibo.remi.lu/
	=> 1.3154006004333496 	 http://repo.manjaro.org.uk/
::Total duration: 1.9168639183044434

Resume all
0.16053271293640137 	 http://ftp.free.org/mirrors/repo.manjaro.org/repos/
0.3029358386993408 	 http://mirror.futureweb.be/manjaro/
0.36032915115356445 	 http://mirror.ragenetwork.de/manjaro/
0.37912607192993164 	 http://mirror.inode.at/manjaro/
0.3841896057128906 	 http://ftp.snt.utwente.nl/pub/linux/manjaro/
0.41837120056152344 	 http://manjaro.mirrors.uk2.net/
0.43140339851379395 	 http://mirror.easyname.at/manjaro/
0.47005653381347656 	 http://ftp.tu-chemnitz.de/pub/linux/manjaro/
0.48391079902648926 	 http://mirror.terrahost.no/linux/manjaro/
0.5288162231445312 	 http://mirror.catn.com/pub/manjaro/
0.5718839168548584 	 https://mirror.23media.com/manjaro/
0.6305742263793945 	 http://mirror.datacenter.by/pub/mirrors/manjaro/
0.6331231594085693 	 http://ftp.vectranet.pl/manjaro/
0.6426975727081299 	 http://mirrors.colocall.net/manjaro/
0.649099588394165 	 https://mirror.neostrada.nl/manjaro/
0.713778018951416 	 https://manjaro.moson.eu/
0.8253636360168457 	 http://ftp.linux.org.tr/manjaro/
0.8285701274871826 	 http://ftp.dei.uc.pt/pub/linux/manjaro/
0.918102502822876 	 http://mirror.truenetwork.ru/manjaro/
0.94211745262146 	 https://quantum-mirror.hu/mirrors/pub/manjaro/
0.9560236930847168 	 https://ftp.caliu.cat/pub/distribucions/manjaro/
1.0911262035369873 	 https://manjaro.grena.ge/
1.128014326095581 	 https://nova.quantum-mirror.hu/mirrors/pub/manjaro/
1.3154006004333496 	 http://repo.manjaro.org.uk/
1.4240171909332275 	 http://kibo.remi.lu/
::Total duration: 1.9168639183044434

Répondre