Commit 0ae4f40d authored by 刘丙寅's avatar 刘丙寅

全上传代码

parent 462e9b29
This diff is collapsed.
This diff is collapsed.
Metadata-Version: 2.1
Name: chardet
Version: 4.0.0
Summary: Universal encoding detector for Python 2 and 3
Home-page: https://github.com/chardet/chardet
Author: Mark Pilgrim
Author-email: mark@diveintomark.org
Maintainer: Daniel Blanchard
Maintainer-email: dan.blanchard@gmail.com
License: LGPL
Keywords: encoding,i18n,xml
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
Chardet: The Universal Character Encoding Detector
--------------------------------------------------
.. image:: https://img.shields.io/travis/chardet/chardet/stable.svg
:alt: Build status
:target: https://travis-ci.org/chardet/chardet
.. image:: https://img.shields.io/coveralls/chardet/chardet/stable.svg
:target: https://coveralls.io/r/chardet/chardet
.. image:: https://img.shields.io/pypi/v/chardet.svg
:target: https://warehouse.python.org/project/chardet/
:alt: Latest version on PyPI
.. image:: https://img.shields.io/pypi/l/chardet.svg
:alt: License
Detects
- ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
- Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
- EUC-JP, SHIFT_JIS, CP932, ISO-2022-JP (Japanese)
- EUC-KR, ISO-2022-KR (Korean)
- KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
- ISO-8859-5, windows-1251 (Bulgarian)
- ISO-8859-1, windows-1252 (Western European languages)
- ISO-8859-7, windows-1253 (Greek)
- ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
- TIS-620 (Thai)
.. note::
Our ISO-8859-2 and windows-1250 (Hungarian) probers have been temporarily
disabled until we can retrain the models.
Requires Python 2.7 or 3.5+.
Installation
------------
Install from `PyPI <https://pypi.org/project/chardet/>`_::
pip install chardet
Documentation
-------------
For users, docs are now available at https://chardet.readthedocs.io/.
Command-line Tool
-----------------
chardet comes with a command-line script which reports on the encodings of one
or more files::
% chardetect somefile someotherfile
somefile: windows-1252 with confidence 0.5
someotherfile: ascii with confidence 1.0
About
-----
This is a continuation of Mark Pilgrim's excellent chardet. Previously, two
versions needed to be maintained: one that supported python 2.x and one that
supported python 3.x. We've recently merged with `Ian Cordasco <https://github.com/sigmavirus24>`_'s
`charade <https://github.com/sigmavirus24/charade>`_ fork, so now we have one
coherent version that works for Python 2.7+ and 3.4+.
:maintainer: Dan Blanchard
chardet/__init__.py,sha256=mWZaWmvZkhwfBEAT9O1Y6nRTfKzhT7FHhQTTAujbqUA,3271
chardet/big5freq.py,sha256=D_zK5GyzoVsRes0HkLJziltFQX0bKCLOrFe9_xDvO_8,31254
chardet/big5prober.py,sha256=kBxHbdetBpPe7xrlb-e990iot64g_eGSLd32lB7_h3M,1757
chardet/chardistribution.py,sha256=3woWS62KrGooKyqz4zQSnjFbJpa6V7g02daAibTwcl8,9411
chardet/charsetgroupprober.py,sha256=GZLReHP6FRRn43hvSOoGCxYamErKzyp6RgOQxVeC3kg,3839
chardet/charsetprober.py,sha256=KSmwJErjypyj0bRZmC5F5eM7c8YQgLYIjZXintZNstg,5110
chardet/codingstatemachine.py,sha256=VYp_6cyyki5sHgXDSZnXW4q1oelHc3cu9AyQTX7uug8,3590
chardet/compat.py,sha256=40zr6wICZwknxyuLGGcIOPyve8DTebBCbbvttvnmp5Q,1200
chardet/cp949prober.py,sha256=TZ434QX8zzBsnUvL_8wm4AQVTZ2ZkqEEQL_lNw9f9ow,1855
chardet/enums.py,sha256=Aimwdb9as1dJKZaFNUH2OhWIVBVd6ZkJJ_WK5sNY8cU,1661
chardet/escprober.py,sha256=kkyqVg1Yw3DIOAMJ2bdlyQgUFQhuHAW8dUGskToNWSc,3950
chardet/escsm.py,sha256=RuXlgNvTIDarndvllNCk5WZBIpdCxQ0kcd9EAuxUh84,10510
chardet/eucjpprober.py,sha256=iD8Jdp0ISRjgjiVN7f0e8xGeQJ5GM2oeZ1dA8nbSeUw,3749
chardet/euckrfreq.py,sha256=-7GdmvgWez4-eO4SuXpa7tBiDi5vRXQ8WvdFAzVaSfo,13546
chardet/euckrprober.py,sha256=MqFMTQXxW4HbzIpZ9lKDHB3GN8SP4yiHenTmf8g_PxY,1748
chardet/euctwfreq.py,sha256=No1WyduFOgB5VITUA7PLyC5oJRNzRyMbBxaKI1l16MA,31621
chardet/euctwprober.py,sha256=13p6EP4yRaxqnP4iHtxHOJ6R2zxHq1_m8hTRjzVZ95c,1747
chardet/gb2312freq.py,sha256=JX8lsweKLmnCwmk8UHEQsLgkr_rP_kEbvivC4qPOrlc,20715
chardet/gb2312prober.py,sha256=gGvIWi9WhDjE-xQXHvNIyrnLvEbMAYgyUSZ65HUfylw,1754
chardet/hebrewprober.py,sha256=c3SZ-K7hvyzGY6JRAZxJgwJ_sUS9k0WYkvMY00YBYFo,13838
chardet/jisfreq.py,sha256=vpmJv2Bu0J8gnMVRPHMFefTRvo_ha1mryLig8CBwgOg,25777
chardet/jpcntx.py,sha256=PYlNqRUQT8LM3cT5FmHGP0iiscFlTWED92MALvBungo,19643
chardet/langbulgarianmodel.py,sha256=r6tvOtO8FqhnbWBB5V4czcl1fWM4pB9lGiWQU-8gvsw,105685
chardet/langgreekmodel.py,sha256=1cMu2wUgPB8bQ2RbVjR4LNwCCETgQ-Dwo0Eg2_uB11s,99559
chardet/langhebrewmodel.py,sha256=urMmJHHIXtCwaWAqy1zEY_4SmwwNzt730bDOtjXzRjs,98764
chardet/langhungarianmodel.py,sha256=ODAisvqCfes8B4FeyM_Pg9HY3ZDnEyaCiT4Bxyzoc6w,102486
chardet/langrussianmodel.py,sha256=sPqkrBbX0QVwwy6oqRl-x7ERv2J4-zaMoCvLpkSsSJI,131168
chardet/langthaimodel.py,sha256=ppoKOGL9OPdj9A4CUyG8R48zbnXt9MN1WXeCYepa6sc,103300
chardet/langturkishmodel.py,sha256=H3ldicI_rhlv0r3VFpVWtUL6X30Wy596v7_YHz2sEdg,95934
chardet/latin1prober.py,sha256=S2IoORhFk39FEFOlSFWtgVybRiP6h7BlLldHVclNkU8,5370
chardet/mbcharsetprober.py,sha256=AR95eFH9vuqSfvLQZN-L5ijea25NOBCoXqw8s5O9xLQ,3413
chardet/mbcsgroupprober.py,sha256=h6TRnnYq2OxG1WdD5JOyxcdVpn7dG0q-vB8nWr5mbh4,2012
chardet/mbcssm.py,sha256=SY32wVIF3HzcjY3BaEspy9metbNSKxIIB0RKPn7tjpI,25481
chardet/sbcharsetprober.py,sha256=nmyMyuxzG87DN6K3Rk2MUzJLMLR69MrWpdnHzOwVUwQ,6136
chardet/sbcsgroupprober.py,sha256=hqefQuXmiFyDBArOjujH6hd6WFXlOD1kWCsxDhjx5Vc,4309
chardet/sjisprober.py,sha256=IIt-lZj0WJqK4rmUZzKZP4GJlE8KUEtFYVuY96ek5MQ,3774
chardet/universaldetector.py,sha256=DpZTXCX0nUHXxkQ9sr4GZxGB_hveZ6hWt3uM94cgWKs,12503
chardet/utf8prober.py,sha256=IdD8v3zWOsB8OLiyPi-y_fqwipRFxV9Nc1eKBLSuIEw,2766
chardet/version.py,sha256=A4CILFAd8MRVG1HoXPp45iK9RLlWyV73a1EtwE8Tvn8,242
chardet/cli/__init__.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
chardet/cli/chardetect.py,sha256=kUPeQCi-olObXpOq5MtlKuBn1EU19rkeenAMwxl7URY,2711
chardet/metadata/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
chardet/metadata/languages.py,sha256=41tLq3eLSrBEbEVVQpVGFq9K7o1ln9b1HpY1l0hCUQo,19474
chardet-4.0.0.dist-info/LICENSE,sha256=YJXp_6d33SKDn3gBqoRbMcntB_PWv4om3F0t7IzMDvM,26432
chardet-4.0.0.dist-info/METADATA,sha256=ySYQAE7NPm3LwxgMqFi1zdLQ48mmwMbrJwqAWCtcbH8,3526
chardet-4.0.0.dist-info/WHEEL,sha256=ADKeyaGyKF5DwBNE0sRE5pvW-bSkFMJfBuhzZ3rceP4,110
chardet-4.0.0.dist-info/entry_points.txt,sha256=fAMmhu5eJ-zAJ-smfqQwRClQ3-nozOCmvJ6-E8lgGJo,60
chardet-4.0.0.dist-info/top_level.txt,sha256=AowzBbZy4x8EirABDdJSLJZMkJ_53iIag8xfKR6D7kI,8
chardet-4.0.0.dist-info/RECORD,,
../../../bin/chardetect,sha256=v8BhfHjGU-xM32L7E6nVxFgEi--nxIBwp6MYeeUOF0k,285
chardet-4.0.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
chardet/__pycache__/mbcharsetprober.cpython-37.pyc,,
chardet/__pycache__/eucjpprober.cpython-37.pyc,,
chardet/__pycache__/langgreekmodel.cpython-37.pyc,,
chardet/__pycache__/sjisprober.cpython-37.pyc,,
chardet/__pycache__/gb2312prober.cpython-37.pyc,,
chardet/__pycache__/euctwfreq.cpython-37.pyc,,
chardet/__pycache__/cp949prober.cpython-37.pyc,,
chardet/__pycache__/version.cpython-37.pyc,,
chardet/__pycache__/langhebrewmodel.cpython-37.pyc,,
chardet/__pycache__/langturkishmodel.cpython-37.pyc,,
chardet/__pycache__/sbcsgroupprober.cpython-37.pyc,,
chardet/__pycache__/euckrfreq.cpython-37.pyc,,
chardet/__pycache__/euckrprober.cpython-37.pyc,,
chardet/__pycache__/escsm.cpython-37.pyc,,
chardet/__pycache__/chardistribution.cpython-37.pyc,,
chardet/__pycache__/euctwprober.cpython-37.pyc,,
chardet/__pycache__/langrussianmodel.cpython-37.pyc,,
chardet/__pycache__/compat.cpython-37.pyc,,
chardet/__pycache__/big5freq.cpython-37.pyc,,
chardet/__pycache__/escprober.cpython-37.pyc,,
chardet/__pycache__/universaldetector.cpython-37.pyc,,
chardet/__pycache__/latin1prober.cpython-37.pyc,,
chardet/__pycache__/hebrewprober.cpython-37.pyc,,
chardet/__pycache__/utf8prober.cpython-37.pyc,,
chardet/__pycache__/langhungarianmodel.cpython-37.pyc,,
chardet/__pycache__/codingstatemachine.cpython-37.pyc,,
chardet/__pycache__/langthaimodel.cpython-37.pyc,,
chardet/__pycache__/enums.cpython-37.pyc,,
chardet/__pycache__/gb2312freq.cpython-37.pyc,,
chardet/__pycache__/big5prober.cpython-37.pyc,,
chardet/__pycache__/mbcssm.cpython-37.pyc,,
chardet/__pycache__/jpcntx.cpython-37.pyc,,
chardet/__pycache__/sbcharsetprober.cpython-37.pyc,,
chardet/__pycache__/jisfreq.cpython-37.pyc,,
chardet/__pycache__/__init__.cpython-37.pyc,,
chardet/__pycache__/charsetprober.cpython-37.pyc,,
chardet/__pycache__/mbcsgroupprober.cpython-37.pyc,,
chardet/__pycache__/langbulgarianmodel.cpython-37.pyc,,
chardet/__pycache__/charsetgroupprober.cpython-37.pyc,,
chardet/cli/__pycache__/chardetect.cpython-37.pyc,,
chardet/cli/__pycache__/__init__.cpython-37.pyc,,
chardet/metadata/__pycache__/__init__.cpython-37.pyc,,
chardet/metadata/__pycache__/languages.cpython-37.pyc,,
Wheel-Version: 1.0
Generator: bdist_wheel (0.35.1)
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
[console_scripts]
chardetect = chardet.cli.chardetect:main
...@@ -16,14 +16,11 @@ ...@@ -16,14 +16,11 @@
######################### END LICENSE BLOCK ######################### ######################### END LICENSE BLOCK #########################
from .compat import PY2, PY3
from .universaldetector import UniversalDetector from .universaldetector import UniversalDetector
from .enums import InputState
from .version import __version__, VERSION from .version import __version__, VERSION
__all__ = ['UniversalDetector', 'detect', 'detect_all', '__version__', 'VERSION']
def detect(byte_str): def detect(byte_str):
""" """
Detect the encoding of the given byte string. Detect the encoding of the given byte string.
...@@ -34,50 +31,9 @@ def detect(byte_str): ...@@ -34,50 +31,9 @@ def detect(byte_str):
if not isinstance(byte_str, bytearray): if not isinstance(byte_str, bytearray):
if not isinstance(byte_str, bytes): if not isinstance(byte_str, bytes):
raise TypeError('Expected object of type bytes or bytearray, got: ' raise TypeError('Expected object of type bytes or bytearray, got: '
'{}'.format(type(byte_str))) '{0}'.format(type(byte_str)))
else: else:
byte_str = bytearray(byte_str) byte_str = bytearray(byte_str)
detector = UniversalDetector() detector = UniversalDetector()
detector.feed(byte_str) detector.feed(byte_str)
return detector.close() return detector.close()
def detect_all(byte_str):
"""
Detect all the possible encodings of the given byte string.
:param byte_str: The byte sequence to examine.
:type byte_str: ``bytes`` or ``bytearray``
"""
if not isinstance(byte_str, bytearray):
if not isinstance(byte_str, bytes):
raise TypeError('Expected object of type bytes or bytearray, got: '
'{}'.format(type(byte_str)))
else:
byte_str = bytearray(byte_str)
detector = UniversalDetector()
detector.feed(byte_str)
detector.close()
if detector._input_state == InputState.HIGH_BYTE:
results = []
for prober in detector._charset_probers:
if prober.get_confidence() > detector.MINIMUM_THRESHOLD:
charset_name = prober.charset_name
lower_charset_name = prober.charset_name.lower()
# Use Windows encoding name instead of ISO-8859 if we saw any
# extra Windows-specific bytes
if lower_charset_name.startswith('iso-8859'):
if detector._has_win_bytes:
charset_name = detector.ISO_WIN_MAP.get(lower_charset_name,
charset_name)
results.append({
'encoding': charset_name,
'confidence': prober.get_confidence(),
'language': prober.language,
})
if len(results) > 0:
return sorted(results, key=lambda result: -result['confidence'])
return [detector.result]
...@@ -73,7 +73,6 @@ class CharSetGroupProber(CharSetProber): ...@@ -73,7 +73,6 @@ class CharSetGroupProber(CharSetProber):
continue continue
if state == ProbingState.FOUND_IT: if state == ProbingState.FOUND_IT:
self._best_guess_prober = prober self._best_guess_prober = prober
self._state = ProbingState.FOUND_IT
return self.state return self.state
elif state == ProbingState.NOT_ME: elif state == ProbingState.NOT_ME:
prober.active = False prober.active = False
......
#!/usr/bin/env python
""" """
Script which takes one or more file paths and reports on their detected Script which takes one or more file paths and reports on their detected
encodings encodings
...@@ -44,10 +45,10 @@ def description_of(lines, name='stdin'): ...@@ -44,10 +45,10 @@ def description_of(lines, name='stdin'):
if PY2: if PY2:
name = name.decode(sys.getfilesystemencoding(), 'ignore') name = name.decode(sys.getfilesystemencoding(), 'ignore')
if result['encoding']: if result['encoding']:
return '{}: {} with confidence {}'.format(name, result['encoding'], return '{0}: {1} with confidence {2}'.format(name, result['encoding'],
result['confidence']) result['confidence'])
else: else:
return '{}: no result'.format(name) return '{0}: no result'.format(name)
def main(argv=None): def main(argv=None):
...@@ -68,7 +69,7 @@ def main(argv=None): ...@@ -68,7 +69,7 @@ def main(argv=None):
type=argparse.FileType('rb'), nargs='*', type=argparse.FileType('rb'), nargs='*',
default=[sys.stdin if PY2 else sys.stdin.buffer]) default=[sys.stdin if PY2 else sys.stdin.buffer])
parser.add_argument('--version', action='version', parser.add_argument('--version', action='version',
version='%(prog)s {}'.format(__version__)) version='%(prog)s {0}'.format(__version__))
args = parser.parse_args(argv) args = parser.parse_args(argv)
for f in args.input: for f in args.input:
......
...@@ -25,12 +25,10 @@ import sys ...@@ -25,12 +25,10 @@ import sys
if sys.version_info < (3, 0): if sys.version_info < (3, 0):
PY2 = True PY2 = True
PY3 = False PY3 = False
string_types = (str, unicode) base_str = (str, unicode)
text_type = unicode text_type = unicode
iteritems = dict.iteritems
else: else:
PY2 = False PY2 = False
PY3 = True PY3 = True
string_types = (bytes, str) base_str = (bytes, str)
text_type = str text_type = str
iteritems = dict.items
...@@ -26,22 +26,10 @@ ...@@ -26,22 +26,10 @@
# 02110-1301 USA # 02110-1301 USA
######################### END LICENSE BLOCK ######################### ######################### END LICENSE BLOCK #########################
from collections import namedtuple
from .charsetprober import CharSetProber from .charsetprober import CharSetProber
from .enums import CharacterCategory, ProbingState, SequenceLikelihood from .enums import CharacterCategory, ProbingState, SequenceLikelihood
SingleByteCharSetModel = namedtuple('SingleByteCharSetModel',
['charset_name',
'language',
'char_to_order_map',
'language_model',
'typical_positive_ratio',
'keep_ascii_letters',
'alphabet'])
class SingleByteCharSetProber(CharSetProber): class SingleByteCharSetProber(CharSetProber):
SAMPLE_SIZE = 64 SAMPLE_SIZE = 64
SB_ENOUGH_REL_THRESHOLD = 1024 # 0.25 * SAMPLE_SIZE^2 SB_ENOUGH_REL_THRESHOLD = 1024 # 0.25 * SAMPLE_SIZE^2
...@@ -77,25 +65,25 @@ class SingleByteCharSetProber(CharSetProber): ...@@ -77,25 +65,25 @@ class SingleByteCharSetProber(CharSetProber):
if self._name_prober: if self._name_prober:
return self._name_prober.charset_name return self._name_prober.charset_name
else: else:
return self._model.charset_name return self._model['charset_name']
@property @property
def language(self): def language(self):
if self._name_prober: if self._name_prober:
return self._name_prober.language return self._name_prober.language
else: else:
return self._model.language return self._model.get('language')
def feed(self, byte_str): def feed(self, byte_str):
# TODO: Make filter_international_words keep things in self.alphabet if not self._model['keep_english_letter']:
if not self._model.keep_ascii_letters:
byte_str = self.filter_international_words(byte_str) byte_str = self.filter_international_words(byte_str)
if not byte_str: if not byte_str:
return self.state return self.state
char_to_order_map = self._model.char_to_order_map char_to_order_map = self._model['char_to_order_map']
language_model = self._model.language_model for i, c in enumerate(byte_str):
for char in byte_str: # XXX: Order is in range 1-64, so one would think we want 0-63 here,
order = char_to_order_map.get(char, CharacterCategory.UNDEFINED) # but that leads to 27 more test failures than before.
order = char_to_order_map[c]
# XXX: This was SYMBOL_CAT_ORDER before, with a value of 250, but # XXX: This was SYMBOL_CAT_ORDER before, with a value of 250, but
# CharacterCategory.SYMBOL is actually 253, so we use CONTROL # CharacterCategory.SYMBOL is actually 253, so we use CONTROL
# to make it closer to the original intent. The only difference # to make it closer to the original intent. The only difference
...@@ -103,21 +91,20 @@ class SingleByteCharSetProber(CharSetProber): ...@@ -103,21 +91,20 @@ class SingleByteCharSetProber(CharSetProber):
# _total_char purposes. # _total_char purposes.
if order < CharacterCategory.CONTROL: if order < CharacterCategory.CONTROL:
self._total_char += 1 self._total_char += 1
# TODO: Follow uchardet's lead and discount confidence for frequent
# control characters.
# See https://github.com/BYVoid/uchardet/commit/55b4f23971db61
if order < self.SAMPLE_SIZE: if order < self.SAMPLE_SIZE:
self._freq_char += 1 self._freq_char += 1
if self._last_order < self.SAMPLE_SIZE: if self._last_order < self.SAMPLE_SIZE:
self._total_seqs += 1 self._total_seqs += 1
if not self._reversed: if not self._reversed:
lm_cat = language_model[self._last_order][order] i = (self._last_order * self.SAMPLE_SIZE) + order
else: model = self._model['precedence_matrix'][i]
lm_cat = language_model[order][self._last_order] else: # reverse the order of the letters in the lookup
self._seq_counters[lm_cat] += 1 i = (order * self.SAMPLE_SIZE) + self._last_order
model = self._model['precedence_matrix'][i]
self._seq_counters[model] += 1
self._last_order = order self._last_order = order
charset_name = self._model.charset_name charset_name = self._model['charset_name']
if self.state == ProbingState.DETECTING: if self.state == ProbingState.DETECTING:
if self._total_seqs > self.SB_ENOUGH_REL_THRESHOLD: if self._total_seqs > self.SB_ENOUGH_REL_THRESHOLD:
confidence = self.get_confidence() confidence = self.get_confidence()
...@@ -138,7 +125,7 @@ class SingleByteCharSetProber(CharSetProber): ...@@ -138,7 +125,7 @@ class SingleByteCharSetProber(CharSetProber):
r = 0.01 r = 0.01
if self._total_seqs > 0: if self._total_seqs > 0:
r = ((1.0 * self._seq_counters[SequenceLikelihood.POSITIVE]) / r = ((1.0 * self._seq_counters[SequenceLikelihood.POSITIVE]) /
self._total_seqs / self._model.typical_positive_ratio) self._total_seqs / self._model['typical_positive_ratio'])
r = r * self._freq_char / self._total_char r = r * self._freq_char / self._total_char
if r >= 1.0: if r >= 1.0:
r = 0.99 r = 0.99
......
...@@ -27,57 +27,47 @@ ...@@ -27,57 +27,47 @@
######################### END LICENSE BLOCK ######################### ######################### END LICENSE BLOCK #########################
from .charsetgroupprober import CharSetGroupProber from .charsetgroupprober import CharSetGroupProber
from .hebrewprober import HebrewProber
from .langbulgarianmodel import (ISO_8859_5_BULGARIAN_MODEL,
WINDOWS_1251_BULGARIAN_MODEL)
from .langgreekmodel import ISO_8859_7_GREEK_MODEL, WINDOWS_1253_GREEK_MODEL
from .langhebrewmodel import WINDOWS_1255_HEBREW_MODEL
# from .langhungarianmodel import (ISO_8859_2_HUNGARIAN_MODEL,
# WINDOWS_1250_HUNGARIAN_MODEL)
from .langrussianmodel import (IBM855_RUSSIAN_MODEL, IBM866_RUSSIAN_MODEL,
ISO_8859_5_RUSSIAN_MODEL, KOI8_R_RUSSIAN_MODEL,
MACCYRILLIC_RUSSIAN_MODEL,
WINDOWS_1251_RUSSIAN_MODEL)
from .langthaimodel import TIS_620_THAI_MODEL
from .langturkishmodel import ISO_8859_9_TURKISH_MODEL
from .sbcharsetprober import SingleByteCharSetProber from .sbcharsetprober import SingleByteCharSetProber
from .langcyrillicmodel import (Win1251CyrillicModel, Koi8rModel,
Latin5CyrillicModel, MacCyrillicModel,
Ibm866Model, Ibm855Model)
from .langgreekmodel import Latin7GreekModel, Win1253GreekModel
from .langbulgarianmodel import Latin5BulgarianModel, Win1251BulgarianModel
# from .langhungarianmodel import Latin2HungarianModel, Win1250HungarianModel
from .langthaimodel import TIS620ThaiModel
from .langhebrewmodel import Win1255HebrewModel
from .hebrewprober import HebrewProber
from .langturkishmodel import Latin5TurkishModel
class SBCSGroupProber(CharSetGroupProber): class SBCSGroupProber(CharSetGroupProber):
def __init__(self): def __init__(self):
super(SBCSGroupProber, self).__init__() super(SBCSGroupProber, self).__init__()
hebrew_prober = HebrewProber()
logical_hebrew_prober = SingleByteCharSetProber(WINDOWS_1255_HEBREW_MODEL,
False, hebrew_prober)
# TODO: See if using ISO-8859-8 Hebrew model works better here, since
# it's actually the visual one
visual_hebrew_prober = SingleByteCharSetProber(WINDOWS_1255_HEBREW_MODEL,
True, hebrew_prober)
hebrew_prober.set_model_probers(logical_hebrew_prober,
visual_hebrew_prober)
# TODO: ORDER MATTERS HERE. I changed the order vs what was in master
# and several tests failed that did not before. Some thought
# should be put into the ordering, and we should consider making
# order not matter here, because that is very counter-intuitive.
self.probers = [ self.probers = [
SingleByteCharSetProber(WINDOWS_1251_RUSSIAN_MODEL), SingleByteCharSetProber(Win1251CyrillicModel),
SingleByteCharSetProber(KOI8_R_RUSSIAN_MODEL), SingleByteCharSetProber(Koi8rModel),
SingleByteCharSetProber(ISO_8859_5_RUSSIAN_MODEL), SingleByteCharSetProber(Latin5CyrillicModel),
SingleByteCharSetProber(MACCYRILLIC_RUSSIAN_MODEL), SingleByteCharSetProber(MacCyrillicModel),
SingleByteCharSetProber(IBM866_RUSSIAN_MODEL), SingleByteCharSetProber(Ibm866Model),
SingleByteCharSetProber(IBM855_RUSSIAN_MODEL), SingleByteCharSetProber(Ibm855Model),
SingleByteCharSetProber(ISO_8859_7_GREEK_MODEL), SingleByteCharSetProber(Latin7GreekModel),
SingleByteCharSetProber(WINDOWS_1253_GREEK_MODEL), SingleByteCharSetProber(Win1253GreekModel),
SingleByteCharSetProber(ISO_8859_5_BULGARIAN_MODEL), SingleByteCharSetProber(Latin5BulgarianModel),
SingleByteCharSetProber(WINDOWS_1251_BULGARIAN_MODEL), SingleByteCharSetProber(Win1251BulgarianModel),
# TODO: Restore Hungarian encodings (iso-8859-2 and windows-1250) # TODO: Restore Hungarian encodings (iso-8859-2 and windows-1250)
# after we retrain model. # after we retrain model.
# SingleByteCharSetProber(ISO_8859_2_HUNGARIAN_MODEL), # SingleByteCharSetProber(Latin2HungarianModel),
# SingleByteCharSetProber(WINDOWS_1250_HUNGARIAN_MODEL), # SingleByteCharSetProber(Win1250HungarianModel),
SingleByteCharSetProber(TIS_620_THAI_MODEL), SingleByteCharSetProber(TIS620ThaiModel),
SingleByteCharSetProber(ISO_8859_9_TURKISH_MODEL), SingleByteCharSetProber(Latin5TurkishModel),
hebrew_prober,
logical_hebrew_prober,
visual_hebrew_prober,
] ]
hebrew_prober = HebrewProber()
logical_hebrew_prober = SingleByteCharSetProber(Win1255HebrewModel,
False, hebrew_prober)
visual_hebrew_prober = SingleByteCharSetProber(Win1255HebrewModel, True,
hebrew_prober)
hebrew_prober.set_model_probers(logical_hebrew_prober, visual_hebrew_prober)
self.probers.extend([hebrew_prober, logical_hebrew_prober,
visual_hebrew_prober])
self.reset() self.reset()
...@@ -266,7 +266,7 @@ class UniversalDetector(object): ...@@ -266,7 +266,7 @@ class UniversalDetector(object):
'language': max_prober.language} 'language': max_prober.language}
# Log all prober confidences if none met MINIMUM_THRESHOLD # Log all prober confidences if none met MINIMUM_THRESHOLD
if self.logger.getEffectiveLevel() <= logging.DEBUG: if self.logger.getEffectiveLevel() == logging.DEBUG:
if self.result['encoding'] is None: if self.result['encoding'] is None:
self.logger.debug('no probers hit minimum threshold') self.logger.debug('no probers hit minimum threshold')
for group_prober in self._charset_probers: for group_prober in self._charset_probers:
...@@ -280,7 +280,7 @@ class UniversalDetector(object): ...@@ -280,7 +280,7 @@ class UniversalDetector(object):
prober.get_confidence()) prober.get_confidence())
else: else:
self.logger.debug('%s %s confidence = %s', self.logger.debug('%s %s confidence = %s',
group_prober.charset_name, prober.charset_name,
group_prober.language, prober.language,
group_prober.get_confidence()) prober.get_confidence())
return self.result return self.result
...@@ -5,5 +5,5 @@ from within setup.py and from chardet subpackages. ...@@ -5,5 +5,5 @@ from within setup.py and from chardet subpackages.
:author: Dan Blanchard (dan.blanchard@gmail.com) :author: Dan Blanchard (dan.blanchard@gmail.com)
""" """
__version__ = "4.0.0" __version__ = "3.0.4"
VERSION = __version__.split('.') VERSION = __version__.split('.')
Metadata-Version: 2.1
Name: requests
Version: 2.25.1
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Project-URL: Documentation, https://requests.readthedocs.io
Project-URL: Source, https://github.com/psf/requests
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
Description-Content-Type: text/markdown
Requires-Dist: chardet (<5,>=3.0.2)
Requires-Dist: idna (<3,>=2.5)
Requires-Dist: urllib3 (<1.27,>=1.21.1)
Requires-Dist: certifi (>=2017.4.17)
Provides-Extra: security
Requires-Dist: pyOpenSSL (>=0.14) ; extra == 'security'
Requires-Dist: cryptography (>=1.3.4) ; extra == 'security'
Provides-Extra: socks
Requires-Dist: PySocks (!=1.5.7,>=1.5.6) ; extra == 'socks'
Requires-Dist: win-inet-pton ; (sys_platform == "win32" and python_version == "2.7") and extra == 'socks'
# Requests
**Requests** is a simple, yet elegant HTTP library.
```python
>>> import requests
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
'{"type":"User"...'
>>> r.json()
{'disk_usage': 368627, 'private_gists': 484, ...}
```
Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your `PUT` & `POST` data — but nowadays, just use the `json` method!
Requests is one of the most downloaded Python package today, pulling in around `14M downloads / week`— according to GitHub, Requests is currently [depended upon](https://github.com/psf/requests/network/dependents?package_id=UGFja2FnZS01NzA4OTExNg%3D%3D) by `500,000+` repositories. You may certainly put your trust in this code.
[![Downloads](https://pepy.tech/badge/requests/month)](https://pepy.tech/project/requests/month)
[![Supported Versions](https://img.shields.io/pypi/pyversions/requests.svg)](https://pypi.org/project/requests)
[![Contributors](https://img.shields.io/github/contributors/psf/requests.svg)](https://github.com/psf/requests/graphs/contributors)
## Installing Requests and Supported Versions
Requests is available on PyPI:
```console
$ python -m pip install requests
```
Requests officially supports Python 2.7 & 3.5+.
## Supported Features & Best–Practices
Requests is ready for the demands of building robust and reliable HTTP–speaking applications, for the needs of today.
- Keep-Alive & Connection Pooling
- International Domains and URLs
- Sessions with Cookie Persistence
- Browser-style TLS/SSL Verification
- Basic & Digest Authentication
- Familiar `dict`–like Cookies
- Automatic Content Decompression and Decoding
- Multi-part File Uploads
- SOCKS Proxy Support
- Connection Timeouts
- Streaming Downloads
- Automatic honoring of `.netrc`
- Chunked HTTP Requests
## API Reference and User Guide available on [Read the Docs](https://requests.readthedocs.io)
[![Read the Docs](https://raw.githubusercontent.com/psf/requests/master/ext/ss.png)](https://requests.readthedocs.io)
---
[![Kenneth Reitz](https://raw.githubusercontent.com/psf/requests/master/ext/kr.png)](https://kennethreitz.org) [![Python Software Foundation](https://raw.githubusercontent.com/psf/requests/master/ext/psf.png)](https://www.python.org/psf)
requests/__init__.py,sha256=rsmg7xmbbCE_zmDcG6EDk_pyvdEfadztdBaWIkInlH8,4141
requests/__version__.py,sha256=k4J8c1yFRFzwGWwlN7miaDOclFtbcIs1GlnmT17YbXQ,441
requests/_internal_utils.py,sha256=Zx3PnEUccyfsB-ie11nZVAW8qClJy0gx1qNME7rgT18,1096
requests/adapters.py,sha256=WelSM1BCQXdbjEuDsBxqKDADeY8BHmxlrwbNnLN2rr4,21344
requests/api.py,sha256=PlHM-HT3PQ5lyufoeGmV-nJxRi7UnUyGVh7OV7B9XV4,6496
requests/auth.py,sha256=OMoJIVKyRLy9THr91y8rxysZuclwPB-K1Xg1zBomUhQ,10207
requests/certs.py,sha256=dOB5rV2DZ13dEhq9BUa_4hd5kAqg59e_zUZB00faYz8,453
requests/compat.py,sha256=iBRvu-X540CH4PJsuxr0vcGTnl_TZhq_75SwmeckQ7w,1782
requests/cookies.py,sha256=Y-bKX6TvW3FnYlE6Au0SXtVVWcaNdFvuAwQxw-G0iTI,18430
requests/exceptions.py,sha256=xXoj1rdhnxTS_DYphKZ9OvFZJQZ333A64REc9ZDZIgU,3161
requests/help.py,sha256=lLcBtKAar8T6T78e9Tc4Zfd_EEJFhntxgib1JHNctEI,3515
requests/hooks.py,sha256=QReGyy0bRcr5rkwCuObNakbYsc7EkiKeBwG4qHekr2Q,757
requests/models.py,sha256=Uhb4Ra_ubNGBf-6ktHShgO5mUSCGZKa5D_wLGVCMtYk,34308
requests/packages.py,sha256=Q2rF0L5mc3wQAvc6q_lAVtPTDOaOeFgD-7kWSQLkjEQ,542
requests/sessions.py,sha256=BsnR-zYILgoFzJ6yq4T8ht_i0PwwPGVAxWxWaV5dcHg,30137
requests/status_codes.py,sha256=gT79Pbs_cQjBgp-fvrUgg1dn2DQO32bDj4TInjnMPSc,4188
requests/structures.py,sha256=msAtr9mq1JxHd-JRyiILfdFlpbJwvvFuP3rfUQT_QxE,3005
requests/utils.py,sha256=_K9AgkN6efPe-a-zgZurXzds5PBC0CzDkyjAE2oCQFQ,30529
requests-2.25.1.dist-info/LICENSE,sha256=CeipvOyAZxBGUsFoaFqwkx54aPnIKEtm9a5u2uXxEws,10142
requests-2.25.1.dist-info/METADATA,sha256=RuNh38uN0IMsRT3OwaTNB_WyGx6RMwwQoMwujXfkUVM,4168
requests-2.25.1.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
requests-2.25.1.dist-info/top_level.txt,sha256=fMSVmHfb5rbGOo6xv-O_tUX6j-WyixssE-SnwcDRxNQ,9
requests-2.25.1.dist-info/RECORD,,
requests-2.25.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
requests/__pycache__/_internal_utils.cpython-37.pyc,,
requests/__pycache__/help.cpython-37.pyc,,
requests/__pycache__/structures.cpython-37.pyc,,
requests/__pycache__/certs.cpython-37.pyc,,
requests/__pycache__/exceptions.cpython-37.pyc,,
requests/__pycache__/__version__.cpython-37.pyc,,
requests/__pycache__/status_codes.cpython-37.pyc,,
requests/__pycache__/sessions.cpython-37.pyc,,
requests/__pycache__/models.cpython-37.pyc,,
requests/__pycache__/compat.cpython-37.pyc,,
requests/__pycache__/adapters.cpython-37.pyc,,
requests/__pycache__/packages.cpython-37.pyc,,
requests/__pycache__/auth.cpython-37.pyc,,
requests/__pycache__/hooks.cpython-37.pyc,,
requests/__pycache__/utils.cpython-37.pyc,,
requests/__pycache__/cookies.cpython-37.pyc,,
requests/__pycache__/api.cpython-37.pyc,,
requests/__pycache__/__init__.cpython-37.pyc,,
Wheel-Version: 1.0
Generator: bdist_wheel (0.36.2)
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
...@@ -57,16 +57,18 @@ def check_compatibility(urllib3_version, chardet_version): ...@@ -57,16 +57,18 @@ def check_compatibility(urllib3_version, chardet_version):
# Check urllib3 for compatibility. # Check urllib3 for compatibility.
major, minor, patch = urllib3_version # noqa: F811 major, minor, patch = urllib3_version # noqa: F811
major, minor, patch = int(major), int(minor), int(patch) major, minor, patch = int(major), int(minor), int(patch)
# urllib3 >= 1.21.1, <= 1.26 # urllib3 >= 1.21.1, <= 1.25
assert major == 1 assert major == 1
assert minor >= 21 assert minor >= 21
assert minor <= 26 assert minor <= 25
# Check chardet for compatibility. # Check chardet for compatibility.
major, minor, patch = chardet_version.split('.')[:3] major, minor, patch = chardet_version.split('.')[:3]
major, minor, patch = int(major), int(minor), int(patch) major, minor, patch = int(major), int(minor), int(patch)
# chardet >= 3.0.2, < 5.0.0 # chardet >= 3.0.2, < 3.1.0
assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0) assert major == 3
assert minor < 1
assert patch >= 2
def _check_cryptography(cryptography_version): def _check_cryptography(cryptography_version):
......
...@@ -5,8 +5,8 @@ ...@@ -5,8 +5,8 @@
__title__ = 'requests' __title__ = 'requests'
__description__ = 'Python HTTP for Humans.' __description__ = 'Python HTTP for Humans.'
__url__ = 'https://requests.readthedocs.io' __url__ = 'https://requests.readthedocs.io'
__version__ = '2.25.1' __version__ = '2.24.0'
__build__ = 0x022501 __build__ = 0x022400
__author__ = 'Kenneth Reitz' __author__ = 'Kenneth Reitz'
__author_email__ = 'me@kennethreitz.org' __author_email__ = 'me@kennethreitz.org'
__license__ = 'Apache 2.0' __license__ = 'Apache 2.0'
......
...@@ -273,9 +273,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin): ...@@ -273,9 +273,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
"""The fully mutable :class:`PreparedRequest <PreparedRequest>` object, """The fully mutable :class:`PreparedRequest <PreparedRequest>` object,
containing the exact bytes that will be sent to the server. containing the exact bytes that will be sent to the server.
Instances are generated from a :class:`Request <Request>` object, and Generated from either a :class:`Request <Request>` object or manually.
should not be instantiated manually; doing so may produce undesirable
effects.
Usage:: Usage::
......
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
requests.sessions requests.session
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
This module provides a Session object to manage and persist settings across This module provides a Session object to manage and persist settings across
requests (cookies, auth, proxies). requests (cookies, auth, proxies).
...@@ -387,13 +387,6 @@ class Session(SessionRedirectMixin): ...@@ -387,13 +387,6 @@ class Session(SessionRedirectMixin):
self.stream = False self.stream = False
#: SSL Verification default. #: SSL Verification default.
#: Defaults to `True`, requiring requests to verify the TLS certificate at the
#: remote end.
#: If verify is set to `False`, requests will accept any TLS certificate
#: presented by the server, and will ignore hostname mismatches and/or
#: expired certificates, which will make your application vulnerable to
#: man-in-the-middle (MitM) attacks.
#: Only set this to `False` for testing.
self.verify = True self.verify = True
#: SSL client certificate default, if String, path to ssl client #: SSL client certificate default, if String, path to ssl client
...@@ -502,12 +495,7 @@ class Session(SessionRedirectMixin): ...@@ -502,12 +495,7 @@ class Session(SessionRedirectMixin):
content. Defaults to ``False``. content. Defaults to ``False``.
:param verify: (optional) Either a boolean, in which case it controls whether we verify :param verify: (optional) Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use. Defaults to ``True``. When set to to a CA bundle to use. Defaults to ``True``.
``False``, requests will accept any TLS certificate presented by
the server, and will ignore hostname mismatches and/or expired
certificates, which will make your application vulnerable to
man-in-the-middle (MitM) attacks. Setting verify to ``False``
may be useful during local development or testing.
:param cert: (optional) if String, path to ssl client cert file (.pem). :param cert: (optional) if String, path to ssl client cert file (.pem).
If Tuple, ('cert', 'key') pair. If Tuple, ('cert', 'key') pair.
:rtype: requests.Response :rtype: requests.Response
......
...@@ -169,20 +169,14 @@ def super_len(o): ...@@ -169,20 +169,14 @@ def super_len(o):
def get_netrc_auth(url, raise_errors=False): def get_netrc_auth(url, raise_errors=False):
"""Returns the Requests tuple auth for a given url from netrc.""" """Returns the Requests tuple auth for a given url from netrc."""
netrc_file = os.environ.get('NETRC')
if netrc_file is not None:
netrc_locations = (netrc_file,)
else:
netrc_locations = ('~/{}'.format(f) for f in NETRC_FILES)
try: try:
from netrc import netrc, NetrcParseError from netrc import netrc, NetrcParseError
netrc_path = None netrc_path = None
for f in netrc_locations: for f in NETRC_FILES:
try: try:
loc = os.path.expanduser(f) loc = os.path.expanduser('~/{}'.format(f))
except KeyError: except KeyError:
# os.path.expanduser can fail when $HOME is undefined and # os.path.expanduser can fail when $HOME is undefined and
# getpwuid fails. See https://bugs.python.org/issue20164 & # getpwuid fails. See https://bugs.python.org/issue20164 &
...@@ -218,7 +212,7 @@ def get_netrc_auth(url, raise_errors=False): ...@@ -218,7 +212,7 @@ def get_netrc_auth(url, raise_errors=False):
if raise_errors: if raise_errors:
raise raise
# App Engine hackiness. # AppEngine hackiness.
except (ImportError, AttributeError): except (ImportError, AttributeError):
pass pass
...@@ -503,10 +497,6 @@ def get_encoding_from_headers(headers): ...@@ -503,10 +497,6 @@ def get_encoding_from_headers(headers):
if 'text' in content_type: if 'text' in content_type:
return 'ISO-8859-1' return 'ISO-8859-1'
if 'application/json' in content_type:
# Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset
return 'utf-8'
def stream_decode_response_unicode(iterator, r): def stream_decode_response_unicode(iterator, r):
"""Stream decodes a iterator.""" """Stream decodes a iterator."""
......
MIT License
Copyright (c) 2008-2020 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
urllib3/__init__.py,sha256=j3yzHIbmW7CS-IKQJ9-PPQf_YKO8EOAey_rMW0UR7us,2763
urllib3/_collections.py,sha256=Rp1mVyBgc_UlAcp6M3at1skJBXR5J43NawRTvW2g_XY,10811
urllib3/_version.py,sha256=nwhnFBUGZnvtqPFpVlkSZM65yja2NkSV_AVO6h9pNfw,63
urllib3/connection.py,sha256=78sb0uVOr87AdmCAUQ7z7BRyAr2y5CGzamEFoKHJeBo,18528
urllib3/connectionpool.py,sha256=IKoeuJZY9YAYm0GK4q-MXAhyXW0M_FnvabYaNsDIR-E,37133
urllib3/exceptions.py,sha256=0Mnno3KHTNfXRfY7638NufOPkUb6mXOm-Lqj-4x2w8A,8217
urllib3/fields.py,sha256=kvLDCg_JmH1lLjUUEY_FLS8UhY7hBvDPuVETbY8mdrM,8579
urllib3/filepost.py,sha256=5b_qqgRHVlL7uLtdAYBzBh-GHmU5AfJVt_2N0XS3PeY,2440
urllib3/poolmanager.py,sha256=whzlX6UTEgODMOCy0ZDMUONRBCz5wyIM8Z9opXAY-Lk,19763
urllib3/request.py,sha256=ZFSIqX0C6WizixecChZ3_okyu7BEv0lZu1VT0s6h4SM,5985
urllib3/response.py,sha256=hGhGBh7TkEkh_IQg5C1W_xuPNrgIKv5BUXPyE-q0LuE,28203
urllib3/contrib/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
urllib3/contrib/_appengine_environ.py,sha256=bDbyOEhW2CKLJcQqAKAyrEHN-aklsyHFKq6vF8ZFsmk,957
urllib3/contrib/appengine.py,sha256=7Pxb0tKfDB_LTGPERiswH0qomhDoUUOo5kwybAKLQyE,11010
urllib3/contrib/ntlmpool.py,sha256=6I95h1_71fzxmoMSNtY0gB8lnyCoVtP_DpqFGj14fdU,4160
urllib3/contrib/pyopenssl.py,sha256=vgh6j52w9xgwq-3R2kfB5M2JblQATJfKAK3lIAc1kSg,16778
urllib3/contrib/securetransport.py,sha256=KxGPZk8d4YepWm7Rc-SBt1XrzIfnLKc8JkUVV75XzgE,34286
urllib3/contrib/socks.py,sha256=DcRjM2l0rQMIyhYrN6r-tnVkY6ZTDxHJlM8_usAkGCA,7097
urllib3/contrib/_securetransport/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
urllib3/contrib/_securetransport/bindings.py,sha256=E1_7ScsgOchfxneozbAueK7ziCwF35fna4DuDCYJ9_o,17637
urllib3/contrib/_securetransport/low_level.py,sha256=lgIdsSycqfB0Xm5BiJzXGeIKT7ybCQMFPJAgkcwPa1s,13908
urllib3/packages/__init__.py,sha256=h4BLhD4tLaBx1adaDtKXfupsgqY0wWLXb_f1_yVlV6A,108
urllib3/packages/six.py,sha256=adx4z-eM_D0Vvu0IIqVzFACQ_ux9l64y7DkSEfbxCDs,32536
urllib3/packages/backports/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
urllib3/packages/backports/makefile.py,sha256=nbzt3i0agPVP07jqqgjhaYjMmuAi_W5E0EywZivVO8E,1417
urllib3/packages/ssl_match_hostname/__init__.py,sha256=zppezdEQdpGsYerI6mV6MfUYy495JV4mcOWC_GgbljU,757
urllib3/packages/ssl_match_hostname/_implementation.py,sha256=6dZ-q074g7XhsJ27MFCgkct8iVNZB3sMZvKhf-KUVy0,5679
urllib3/util/__init__.py,sha256=JEmSmmqqLyaw8P51gUImZh8Gwg9i1zSe-DoqAitn2nc,1155
urllib3/util/connection.py,sha256=21B-LX0c8fkxPDssyHCaK0pCnmrKmhltg5EoouHiAPU,4910
urllib3/util/proxy.py,sha256=FGipAEnvZteyldXNjce4DEB7YzwU-a5lep8y5S0qHQg,1604
urllib3/util/queue.py,sha256=nRgX8_eX-_VkvxoX096QWoz8Ps0QHUAExILCY_7PncM,498
urllib3/util/request.py,sha256=NnzaEKQ1Pauw5MFMV6HmgEMHITf0Aua9fQuzi2uZzGc,4123
urllib3/util/response.py,sha256=GJpg3Egi9qaJXRwBh5wv-MNuRWan5BIu40oReoxWP28,3510
urllib3/util/retry.py,sha256=tn168HDMUynFmXRP-uVaLRUOlbTEJikoB1RuZdwfCes,21366
urllib3/util/ssl_.py,sha256=fRqzRmoLfiDv5_tFzgFtGzlOuuHoLX0us7WUf7tYDAo,16269
urllib3/util/ssltransport.py,sha256=IvGQvs9YWkf4jzfqVjTu_UWjwAUgPn5ActajW8VLz6A,6908
urllib3/util/timeout.py,sha256=QSbBUNOB9yh6AnDn61SrLQ0hg5oz0I9-uXEG91AJuIg,10003
urllib3/util/url.py,sha256=LWfLSlI4l2FmUMKfCkElCaW10-0N-sJDT9bxaDZJkjs,13964
urllib3/util/wait.py,sha256=3MUKRSAUJDB2tgco7qRUskW0zXGAWYvRRE4Q1_6xlLs,5404
urllib3-1.26.3.dist-info/LICENSE.txt,sha256=w3vxhuJ8-dvpYZ5V7f486nswCRzrPaY8fay-Dm13kHs,1115
urllib3-1.26.3.dist-info/METADATA,sha256=bfUq23qlxCvoPweVN1YGCu-yeErYC5EbVCmF6tUG4Qs,43231
urllib3-1.26.3.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
urllib3-1.26.3.dist-info/top_level.txt,sha256=EMiXL2sKrTcmrMxIHTqdc3ET54pQI2Y072LexFEemvo,8
urllib3-1.26.3.dist-info/RECORD,,
urllib3-1.26.3.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
urllib3/util/__pycache__/response.cpython-37.pyc,,
urllib3/util/__pycache__/timeout.cpython-37.pyc,,
urllib3/util/__pycache__/proxy.cpython-37.pyc,,
urllib3/util/__pycache__/url.cpython-37.pyc,,
urllib3/util/__pycache__/request.cpython-37.pyc,,
urllib3/util/__pycache__/connection.cpython-37.pyc,,
urllib3/util/__pycache__/wait.cpython-37.pyc,,
urllib3/util/__pycache__/ssltransport.cpython-37.pyc,,
urllib3/util/__pycache__/ssl_.cpython-37.pyc,,
urllib3/util/__pycache__/retry.cpython-37.pyc,,
urllib3/util/__pycache__/__init__.cpython-37.pyc,,
urllib3/util/__pycache__/queue.cpython-37.pyc,,
urllib3/__pycache__/response.cpython-37.pyc,,
urllib3/__pycache__/poolmanager.cpython-37.pyc,,
urllib3/__pycache__/_version.cpython-37.pyc,,
urllib3/__pycache__/exceptions.cpython-37.pyc,,
urllib3/__pycache__/request.cpython-37.pyc,,
urllib3/__pycache__/filepost.cpython-37.pyc,,
urllib3/__pycache__/connectionpool.cpython-37.pyc,,
urllib3/__pycache__/connection.cpython-37.pyc,,
urllib3/__pycache__/_collections.cpython-37.pyc,,
urllib3/__pycache__/fields.cpython-37.pyc,,
urllib3/__pycache__/__init__.cpython-37.pyc,,
urllib3/contrib/__pycache__/ntlmpool.cpython-37.pyc,,
urllib3/contrib/__pycache__/securetransport.cpython-37.pyc,,
urllib3/contrib/__pycache__/appengine.cpython-37.pyc,,
urllib3/contrib/__pycache__/socks.cpython-37.pyc,,
urllib3/contrib/__pycache__/_appengine_environ.cpython-37.pyc,,
urllib3/contrib/__pycache__/pyopenssl.cpython-37.pyc,,
urllib3/contrib/__pycache__/__init__.cpython-37.pyc,,
urllib3/contrib/_securetransport/__pycache__/low_level.cpython-37.pyc,,
urllib3/contrib/_securetransport/__pycache__/bindings.cpython-37.pyc,,
urllib3/contrib/_securetransport/__pycache__/__init__.cpython-37.pyc,,
urllib3/packages/__pycache__/six.cpython-37.pyc,,
urllib3/packages/__pycache__/__init__.cpython-37.pyc,,
urllib3/packages/backports/__pycache__/makefile.cpython-37.pyc,,
urllib3/packages/backports/__pycache__/__init__.cpython-37.pyc,,
urllib3/packages/ssl_match_hostname/__pycache__/_implementation.cpython-37.pyc,,
urllib3/packages/ssl_match_hostname/__pycache__/__init__.cpython-37.pyc,,
Wheel-Version: 1.0
Generator: bdist_wheel (0.36.2)
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
""" """
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more urllib3 - Thread-safe connection pooling and re-using.
""" """
from __future__ import absolute_import from __future__ import absolute_import
# Set default logging handler to avoid "No handler found" warnings.
import logging
import warnings import warnings
from logging import NullHandler
from . import exceptions
from ._version import __version__
from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
from . import exceptions
from .filepost import encode_multipart_formdata from .filepost import encode_multipart_formdata
from .poolmanager import PoolManager, ProxyManager, proxy_from_url from .poolmanager import PoolManager, ProxyManager, proxy_from_url
from .response import HTTPResponse from .response import HTTPResponse
from .util.request import make_headers from .util.request import make_headers
from .util.retry import Retry
from .util.timeout import Timeout
from .util.url import get_host from .util.url import get_host
from .util.timeout import Timeout
from .util.retry import Retry
from ._version import __version__
# Set default logging handler to avoid "No handler found" warnings.
import logging
from logging import NullHandler
__author__ = "Andrey Petrov (andrey.petrov@shazow.net)" __author__ = "Andrey Petrov (andrey.petrov@shazow.net)"
__license__ = "MIT" __license__ = "MIT"
......
...@@ -17,10 +17,9 @@ except ImportError: # Platform-specific: No threads available ...@@ -17,10 +17,9 @@ except ImportError: # Platform-specific: No threads available
from collections import OrderedDict from collections import OrderedDict
from .exceptions import InvalidHeader from .exceptions import InvalidHeader
from .packages import six from .packages.six import iterkeys, itervalues, PY3
from .packages.six import iterkeys, itervalues
__all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"] __all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"]
...@@ -175,7 +174,7 @@ class HTTPHeaderDict(MutableMapping): ...@@ -175,7 +174,7 @@ class HTTPHeaderDict(MutableMapping):
def __ne__(self, other): def __ne__(self, other):
return not self.__eq__(other) return not self.__eq__(other)
if six.PY2: # Python 2 if not PY3: # Python 2
iterkeys = MutableMapping.iterkeys iterkeys = MutableMapping.iterkeys
itervalues = MutableMapping.itervalues itervalues = MutableMapping.itervalues
......
# This file is protected via CODEOWNERS # This file is protected via CODEOWNERS
__version__ = "1.26.3" __version__ = "1.25.11"
...@@ -32,24 +32,22 @@ license and by oscrypto's: ...@@ -32,24 +32,22 @@ license and by oscrypto's:
from __future__ import absolute_import from __future__ import absolute_import
import platform import platform
from ctypes.util import find_library
from ctypes import ( from ctypes import (
CDLL, c_void_p,
CFUNCTYPE,
POINTER,
c_bool,
c_byte,
c_char_p,
c_int32, c_int32,
c_long, c_char_p,
c_size_t, c_size_t,
c_byte,
c_uint32, c_uint32,
c_ulong, c_ulong,
c_void_p, c_long,
c_bool,
) )
from ctypes.util import find_library from ctypes import CDLL, POINTER, CFUNCTYPE
from urllib3.packages.six import raise_from from urllib3.packages.six import raise_from
if platform.system() != "Darwin": if platform.system() != "Darwin":
raise ImportError("Only macOS is supported") raise ImportError("Only macOS is supported")
...@@ -295,13 +293,6 @@ try: ...@@ -295,13 +293,6 @@ try:
Security.SSLSetProtocolVersionMax.argtypes = [SSLContextRef, SSLProtocol] Security.SSLSetProtocolVersionMax.argtypes = [SSLContextRef, SSLProtocol]
Security.SSLSetProtocolVersionMax.restype = OSStatus Security.SSLSetProtocolVersionMax.restype = OSStatus
try:
Security.SSLSetALPNProtocols.argtypes = [SSLContextRef, CFArrayRef]
Security.SSLSetALPNProtocols.restype = OSStatus
except AttributeError:
# Supported only in 10.12+
pass
Security.SecCopyErrorMessageString.argtypes = [OSStatus, c_void_p] Security.SecCopyErrorMessageString.argtypes = [OSStatus, c_void_p]
Security.SecCopyErrorMessageString.restype = CFStringRef Security.SecCopyErrorMessageString.restype = CFStringRef
......
...@@ -10,13 +10,13 @@ appropriate and useful assistance to the higher-level code. ...@@ -10,13 +10,13 @@ appropriate and useful assistance to the higher-level code.
import base64 import base64
import ctypes import ctypes
import itertools import itertools
import os
import re import re
import os
import ssl import ssl
import struct
import tempfile import tempfile
from .bindings import CFConst, CoreFoundation, Security from .bindings import Security, CoreFoundation, CFConst
# This regular expression is used to grab PEM data out of a PEM bundle. # This regular expression is used to grab PEM data out of a PEM bundle.
_PEM_CERTS_RE = re.compile( _PEM_CERTS_RE = re.compile(
...@@ -56,51 +56,6 @@ def _cf_dictionary_from_tuples(tuples): ...@@ -56,51 +56,6 @@ def _cf_dictionary_from_tuples(tuples):
) )
def _cfstr(py_bstr):
"""
Given a Python binary data, create a CFString.
The string must be CFReleased by the caller.
"""
c_str = ctypes.c_char_p(py_bstr)
cf_str = CoreFoundation.CFStringCreateWithCString(
CoreFoundation.kCFAllocatorDefault,
c_str,
CFConst.kCFStringEncodingUTF8,
)
return cf_str
def _create_cfstring_array(lst):
"""
Given a list of Python binary data, create an associated CFMutableArray.
The array must be CFReleased by the caller.
Raises an ssl.SSLError on failure.
"""
cf_arr = None
try:
cf_arr = CoreFoundation.CFArrayCreateMutable(
CoreFoundation.kCFAllocatorDefault,
0,
ctypes.byref(CoreFoundation.kCFTypeArrayCallBacks),
)
if not cf_arr:
raise MemoryError("Unable to allocate memory!")
for item in lst:
cf_str = _cfstr(item)
if not cf_str:
raise MemoryError("Unable to allocate memory!")
try:
CoreFoundation.CFArrayAppendValue(cf_arr, cf_str)
finally:
CoreFoundation.CFRelease(cf_str)
except BaseException as e:
if cf_arr:
CoreFoundation.CFRelease(cf_arr)
raise ssl.SSLError("Unable to allocate array: %s" % (e,))
return cf_arr
def _cf_string_to_unicode(value): def _cf_string_to_unicode(value):
""" """
Creates a Unicode string from a CFString object. Used entirely for error Creates a Unicode string from a CFString object. Used entirely for error
...@@ -371,26 +326,3 @@ def _load_client_cert_chain(keychain, *paths): ...@@ -371,26 +326,3 @@ def _load_client_cert_chain(keychain, *paths):
finally: finally:
for obj in itertools.chain(identities, certificates): for obj in itertools.chain(identities, certificates):
CoreFoundation.CFRelease(obj) CoreFoundation.CFRelease(obj)
TLS_PROTOCOL_VERSIONS = {
"SSLv2": (0, 2),
"SSLv3": (3, 0),
"TLSv1": (3, 1),
"TLSv1.1": (3, 2),
"TLSv1.2": (3, 3),
}
def _build_tls_unknown_ca_alert(version):
"""
Builds a TLS alert record for an unknown CA.
"""
ver_maj, ver_min = TLS_PROTOCOL_VERSIONS[version]
severity_fatal = 0x02
description_unknown_ca = 0x30
msg = struct.pack(">BB", severity_fatal, description_unknown_ca)
msg_len = len(msg)
record_type_alert = 0x15
record = struct.pack(">BBBH", record_type_alert, ver_maj, ver_min, msg_len) + msg
return record
...@@ -39,24 +39,24 @@ urllib3 on Google App Engine: ...@@ -39,24 +39,24 @@ urllib3 on Google App Engine:
""" """
from __future__ import absolute_import from __future__ import absolute_import
import io import io
import logging import logging
import warnings import warnings
from ..packages.six.moves.urllib.parse import urljoin
from ..exceptions import ( from ..exceptions import (
HTTPError, HTTPError,
HTTPWarning, HTTPWarning,
MaxRetryError, MaxRetryError,
ProtocolError, ProtocolError,
SSLError,
TimeoutError, TimeoutError,
SSLError,
) )
from ..packages.six.moves.urllib.parse import urljoin
from ..request import RequestMethods from ..request import RequestMethods
from ..response import HTTPResponse from ..response import HTTPResponse
from ..util.retry import Retry
from ..util.timeout import Timeout from ..util.timeout import Timeout
from ..util.retry import Retry
from . import _appengine_environ from . import _appengine_environ
try: try:
...@@ -90,7 +90,7 @@ class AppEngineManager(RequestMethods): ...@@ -90,7 +90,7 @@ class AppEngineManager(RequestMethods):
* If you attempt to use this on App Engine Flexible, as full socket * If you attempt to use this on App Engine Flexible, as full socket
support is available. support is available.
* If a request size is more than 10 megabytes. * If a request size is more than 10 megabytes.
* If a response size is more than 32 megabytes. * If a response size is more than 32 megabtyes.
* If you use an unsupported request method such as OPTIONS. * If you use an unsupported request method such as OPTIONS.
Beyond those cases, it will raise normal urllib3 errors. Beyond those cases, it will raise normal urllib3 errors.
......
...@@ -6,12 +6,12 @@ Issue #10, see: http://code.google.com/p/urllib3/issues/detail?id=10 ...@@ -6,12 +6,12 @@ Issue #10, see: http://code.google.com/p/urllib3/issues/detail?id=10
from __future__ import absolute_import from __future__ import absolute_import
from logging import getLogger from logging import getLogger
from ntlm import ntlm from ntlm import ntlm
from .. import HTTPSConnectionPool from .. import HTTPSConnectionPool
from ..packages.six.moves.http_client import HTTPSConnection from ..packages.six.moves.http_client import HTTPSConnection
log = getLogger(__name__) log = getLogger(__name__)
......
""" """
TLS with SNI_-support for Python 2. Follow these instructions if you would SSL with SNI_-support for Python 2. Follow these instructions if you would
like to verify TLS certificates in Python 2. Note, the default libraries do like to verify SSL certificates in Python 2. Note, the default libraries do
*not* do certificate checking; you need to do additional work to validate *not* do certificate checking; you need to do additional work to validate
certificates yourself. certificates yourself.
This needs the following packages installed: This needs the following packages installed:
* `pyOpenSSL`_ (tested with 16.0.0) * pyOpenSSL (tested with 16.0.0)
* `cryptography`_ (minimum 1.3.4, from pyopenssl) * cryptography (minimum 1.3.4, from pyopenssl)
* `idna`_ (minimum 2.0, from cryptography) * idna (minimum 2.0, from cryptography)
However, pyopenssl depends on cryptography, which depends on idna, so while we However, pyopenssl depends on cryptography, which depends on idna, so while we
use all three directly here we end up having relatively few packages required. use all three directly here we end up having relatively few packages required.
You can install them with the following command: You can install them with the following command:
.. code-block:: bash pip install pyopenssl cryptography idna
$ python -m pip install pyopenssl cryptography idna
To activate certificate checking, call To activate certificate checking, call
:func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code :func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code
before you begin making HTTP requests. This can be done in a ``sitecustomize`` before you begin making HTTP requests. This can be done in a ``sitecustomize``
module, or at any other time before your application begins using ``urllib3``, module, or at any other time before your application begins using ``urllib3``,
like this: like this::
.. code-block:: python
try: try:
import urllib3.contrib.pyopenssl import urllib3.contrib.pyopenssl
...@@ -39,11 +35,11 @@ when the required modules are installed. ...@@ -39,11 +35,11 @@ when the required modules are installed.
Activating this module also has the positive side effect of disabling SSL/TLS Activating this module also has the positive side effect of disabling SSL/TLS
compression in Python 2 (see `CRIME attack`_). compression in Python 2 (see `CRIME attack`_).
If you want to configure the default list of supported cipher suites, you can
set the ``urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST`` variable.
.. _sni: https://en.wikipedia.org/wiki/Server_Name_Indication .. _sni: https://en.wikipedia.org/wiki/Server_Name_Indication
.. _crime attack: https://en.wikipedia.org/wiki/CRIME_(security_exploit) .. _crime attack: https://en.wikipedia.org/wiki/CRIME_(security_exploit)
.. _pyopenssl: https://www.pyopenssl.org
.. _cryptography: https://cryptography.io
.. _idna: https://github.com/kjd/idna
""" """
from __future__ import absolute_import from __future__ import absolute_import
...@@ -60,9 +56,8 @@ except ImportError: ...@@ -60,9 +56,8 @@ except ImportError:
pass pass
from socket import timeout, error as SocketError
from io import BytesIO from io import BytesIO
from socket import error as SocketError
from socket import timeout
try: # Platform-specific: Python 2 try: # Platform-specific: Python 2
from socket import _fileobject from socket import _fileobject
...@@ -72,10 +67,11 @@ except ImportError: # Platform-specific: Python 3 ...@@ -72,10 +67,11 @@ except ImportError: # Platform-specific: Python 3
import logging import logging
import ssl import ssl
from ..packages import six
import sys import sys
from .. import util from .. import util
from ..packages import six
__all__ = ["inject_into_urllib3", "extract_from_urllib3"] __all__ = ["inject_into_urllib3", "extract_from_urllib3"]
...@@ -469,10 +465,6 @@ class PyOpenSSLContext(object): ...@@ -469,10 +465,6 @@ class PyOpenSSLContext(object):
self._ctx.set_passwd_cb(lambda *_: password) self._ctx.set_passwd_cb(lambda *_: password)
self._ctx.use_privatekey_file(keyfile or certfile) self._ctx.use_privatekey_file(keyfile or certfile)
def set_alpn_protocols(self, protocols):
protocols = [six.ensure_binary(p) for p in protocols]
return self._ctx.set_alpn_protos(protocols)
def wrap_socket( def wrap_socket(
self, self,
sock, sock,
......
...@@ -29,8 +29,6 @@ library. An enormous debt is owed to him for blazing this trail for us. For ...@@ -29,8 +29,6 @@ library. An enormous debt is owed to him for blazing this trail for us. For
that reason, this code should be considered to be covered both by urllib3's that reason, this code should be considered to be covered both by urllib3's
license and by oscrypto's: license and by oscrypto's:
.. code-block::
Copyright (c) 2015-2016 Will Bond <will@wbond.net> Copyright (c) 2015-2016 Will Bond <will@wbond.net>
Permission is hereby granted, free of charge, to any person obtaining a Permission is hereby granted, free of charge, to any person obtaining a
...@@ -60,21 +58,16 @@ import os.path ...@@ -60,21 +58,16 @@ import os.path
import shutil import shutil
import socket import socket
import ssl import ssl
import struct
import threading import threading
import weakref import weakref
import six
from .. import util from .. import util
from ._securetransport.bindings import CoreFoundation, Security, SecurityConst from ._securetransport.bindings import Security, SecurityConst, CoreFoundation
from ._securetransport.low_level import ( from ._securetransport.low_level import (
_assert_no_error, _assert_no_error,
_build_tls_unknown_ca_alert,
_cert_array_from_pem, _cert_array_from_pem,
_create_cfstring_array,
_load_client_cert_chain,
_temporary_keychain, _temporary_keychain,
_load_client_cert_chain,
) )
try: # Platform-specific: Python 2 try: # Platform-specific: Python 2
...@@ -381,55 +374,16 @@ class WrappedSocket(object): ...@@ -381,55 +374,16 @@ class WrappedSocket(object):
) )
_assert_no_error(result) _assert_no_error(result)
def _set_alpn_protocols(self, protocols):
"""
Sets up the ALPN protocols on the context.
"""
if not protocols:
return
protocols_arr = _create_cfstring_array(protocols)
try:
result = Security.SSLSetALPNProtocols(self.context, protocols_arr)
_assert_no_error(result)
finally:
CoreFoundation.CFRelease(protocols_arr)
def _custom_validate(self, verify, trust_bundle): def _custom_validate(self, verify, trust_bundle):
""" """
Called when we have set custom validation. We do this in two cases: Called when we have set custom validation. We do this in two cases:
first, when cert validation is entirely disabled; and second, when first, when cert validation is entirely disabled; and second, when
using a custom trust DB. using a custom trust DB.
Raises an SSLError if the connection is not trusted.
""" """
# If we disabled cert validation, just say: cool. # If we disabled cert validation, just say: cool.
if not verify: if not verify:
return return
successes = (
SecurityConst.kSecTrustResultUnspecified,
SecurityConst.kSecTrustResultProceed,
)
try:
trust_result = self._evaluate_trust(trust_bundle)
if trust_result in successes:
return
reason = "error code: %d" % (trust_result,)
except Exception as e:
# Do not trust on error
reason = "exception: %r" % (e,)
# SecureTransport does not send an alert nor shuts down the connection.
rec = _build_tls_unknown_ca_alert(self.version())
self.socket.sendall(rec)
# close the connection immediately
# l_onoff = 1, activate linger
# l_linger = 0, linger for 0 seoncds
opts = struct.pack("ii", 1, 0)
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, opts)
self.close()
raise ssl.SSLError("certificate verify failed, %s" % reason)
def _evaluate_trust(self, trust_bundle):
# We want data in memory, so load it up. # We want data in memory, so load it up.
if os.path.isfile(trust_bundle): if os.path.isfile(trust_bundle):
with open(trust_bundle, "rb") as f: with open(trust_bundle, "rb") as f:
...@@ -467,7 +421,15 @@ class WrappedSocket(object): ...@@ -467,7 +421,15 @@ class WrappedSocket(object):
if cert_array is not None: if cert_array is not None:
CoreFoundation.CFRelease(cert_array) CoreFoundation.CFRelease(cert_array)
return trust_result.value # Ok, now we can look at what the result was.
successes = (
SecurityConst.kSecTrustResultUnspecified,
SecurityConst.kSecTrustResultProceed,
)
if trust_result.value not in successes:
raise ssl.SSLError(
"certificate verify failed, error code: %d" % trust_result.value
)
def handshake( def handshake(
self, self,
...@@ -479,7 +441,6 @@ class WrappedSocket(object): ...@@ -479,7 +441,6 @@ class WrappedSocket(object):
client_cert, client_cert,
client_key, client_key,
client_key_passphrase, client_key_passphrase,
alpn_protocols,
): ):
""" """
Actually performs the TLS handshake. This is run automatically by Actually performs the TLS handshake. This is run automatically by
...@@ -520,9 +481,6 @@ class WrappedSocket(object): ...@@ -520,9 +481,6 @@ class WrappedSocket(object):
# Setup the ciphers. # Setup the ciphers.
self._set_ciphers() self._set_ciphers()
# Setup the ALPN protocols.
self._set_alpn_protocols(alpn_protocols)
# Set the minimum and maximum TLS versions. # Set the minimum and maximum TLS versions.
result = Security.SSLSetProtocolVersionMin(self.context, min_version) result = Security.SSLSetProtocolVersionMin(self.context, min_version)
_assert_no_error(result) _assert_no_error(result)
...@@ -796,7 +754,6 @@ class SecureTransportContext(object): ...@@ -796,7 +754,6 @@ class SecureTransportContext(object):
self._client_cert = None self._client_cert = None
self._client_key = None self._client_key = None
self._client_key_passphrase = None self._client_key_passphrase = None
self._alpn_protocols = None
@property @property
def check_hostname(self): def check_hostname(self):
...@@ -874,18 +831,6 @@ class SecureTransportContext(object): ...@@ -874,18 +831,6 @@ class SecureTransportContext(object):
self._client_key = keyfile self._client_key = keyfile
self._client_cert_passphrase = password self._client_cert_passphrase = password
def set_alpn_protocols(self, protocols):
"""
Sets the ALPN protocols that will later be set on the context.
Raises a NotImplementedError if ALPN is not supported.
"""
if not hasattr(Security, "SSLSetALPNProtocols"):
raise NotImplementedError(
"SecureTransport supports ALPN only in macOS 10.12+"
)
self._alpn_protocols = [six.ensure_binary(p) for p in protocols]
def wrap_socket( def wrap_socket(
self, self,
sock, sock,
...@@ -915,6 +860,5 @@ class SecureTransportContext(object): ...@@ -915,6 +860,5 @@ class SecureTransportContext(object):
self._client_cert, self._client_cert,
self._client_key, self._client_key,
self._client_key_passphrase, self._client_key_passphrase,
self._alpn_protocols,
) )
return wrapped_socket return wrapped_socket
...@@ -14,7 +14,7 @@ supports the following SOCKS features: ...@@ -14,7 +14,7 @@ supports the following SOCKS features:
- SOCKS5 with local DNS (``proxy_url='socks5://...``) - SOCKS5 with local DNS (``proxy_url='socks5://...``)
- Usernames and passwords for the SOCKS proxy - Usernames and passwords for the SOCKS proxy
.. note:: .. note::
It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in
your ``proxy_url`` to ensure that DNS resolution is done from the remote your ``proxy_url`` to ensure that DNS resolution is done from the remote
server instead of client-side when connecting to a domain name. server instead of client-side when connecting to a domain name.
...@@ -23,17 +23,13 @@ SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5 ...@@ -23,17 +23,13 @@ SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5
supports IPv4, IPv6, and domain names. supports IPv4, IPv6, and domain names.
When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url`` When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url``
will be sent as the ``userid`` section of the SOCKS request: will be sent as the ``userid`` section of the SOCKS request::
.. code-block:: python
proxy_url="socks4a://<userid>@proxy-host" proxy_url="socks4a://<userid>@proxy-host"
When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion
of the ``proxy_url`` will be sent as the username/password to authenticate of the ``proxy_url`` will be sent as the username/password to authenticate
with the proxy: with the proxy::
.. code-block:: python
proxy_url="socks5h://<username>:<password>@proxy-host" proxy_url="socks5h://<username>:<password>@proxy-host"
...@@ -44,7 +40,6 @@ try: ...@@ -44,7 +40,6 @@ try:
import socks import socks
except ImportError: except ImportError:
import warnings import warnings
from ..exceptions import DependencyWarning from ..exceptions import DependencyWarning
warnings.warn( warnings.warn(
...@@ -57,8 +52,7 @@ except ImportError: ...@@ -57,8 +52,7 @@ except ImportError:
) )
raise raise
from socket import error as SocketError from socket import error as SocketError, timeout as SocketTimeout
from socket import timeout as SocketTimeout
from ..connection import HTTPConnection, HTTPSConnection from ..connection import HTTPConnection, HTTPSConnection
from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool
......
from __future__ import absolute_import from __future__ import absolute_import
import email.utils import email.utils
import mimetypes import mimetypes
import re import re
...@@ -27,8 +26,7 @@ def format_header_param_rfc2231(name, value): ...@@ -27,8 +26,7 @@ def format_header_param_rfc2231(name, value):
strategy defined in RFC 2231. strategy defined in RFC 2231.
Particularly useful for header parameters which might contain Particularly useful for header parameters which might contain
non-ASCII values, like file names. This follows non-ASCII values, like file names. This follows RFC 2388 Section 4.4.
`RFC 2388 Section 4.4 <https://tools.ietf.org/html/rfc2388#section-4.4>`_.
:param name: :param name:
The name of the parameter, a string expected to be ASCII only. The name of the parameter, a string expected to be ASCII only.
...@@ -67,6 +65,7 @@ _HTML5_REPLACEMENTS = { ...@@ -67,6 +65,7 @@ _HTML5_REPLACEMENTS = {
u"\u0022": u"%22", u"\u0022": u"%22",
# Replace "\" with "\\". # Replace "\" with "\\".
u"\u005C": u"\u005C\u005C", u"\u005C": u"\u005C\u005C",
u"\u005C": u"\u005C\u005C",
} }
# All control characters from 0x00 to 0x1F *except* 0x1B. # All control characters from 0x00 to 0x1F *except* 0x1B.
......
from __future__ import absolute_import from __future__ import absolute_import
import binascii import binascii
import codecs import codecs
import os import os
from io import BytesIO from io import BytesIO
from .fields import RequestField
from .packages import six from .packages import six
from .packages.six import b from .packages.six import b
from .fields import RequestField
writer = codecs.lookup("utf-8")[3] writer = codecs.lookup("utf-8")[3]
......
...@@ -7,6 +7,7 @@ Backports the Python 3 ``socket.makefile`` method for use with anything that ...@@ -7,6 +7,7 @@ Backports the Python 3 ``socket.makefile`` method for use with anything that
wants to create a "fake" socket object. wants to create a "fake" socket object.
""" """
import io import io
from socket import SocketIO from socket import SocketIO
......
...@@ -10,13 +10,10 @@ try: ...@@ -10,13 +10,10 @@ try:
except ImportError: except ImportError:
try: try:
# Backport of the function from a pypi module # Backport of the function from a pypi module
from backports.ssl_match_hostname import ( # type: ignore from backports.ssl_match_hostname import CertificateError, match_hostname
CertificateError,
match_hostname,
)
except ImportError: except ImportError:
# Our vendored copy # Our vendored copy
from ._implementation import CertificateError, match_hostname # type: ignore from ._implementation import CertificateError, match_hostname
# Not needed, but documenting what we provide. # Not needed, but documenting what we provide.
__all__ = ("CertificateError", "match_hostname") __all__ = ("CertificateError", "match_hostname")
...@@ -3,14 +3,15 @@ from __future__ import absolute_import ...@@ -3,14 +3,15 @@ from __future__ import absolute_import
from .filepost import encode_multipart_formdata from .filepost import encode_multipart_formdata
from .packages.six.moves.urllib.parse import urlencode from .packages.six.moves.urllib.parse import urlencode
__all__ = ["RequestMethods"] __all__ = ["RequestMethods"]
class RequestMethods(object): class RequestMethods(object):
""" """
Convenience mixin for classes who implement a :meth:`urlopen` method, such Convenience mixin for classes who implement a :meth:`urlopen` method, such
as :class:`urllib3.HTTPConnectionPool` and as :class:`~urllib3.connectionpool.HTTPConnectionPool` and
:class:`urllib3.PoolManager`. :class:`~urllib3.poolmanager.PoolManager`.
Provides behavior for making common types of HTTP request methods and Provides behavior for making common types of HTTP request methods and
decides which type of request field encoding to use. decides which type of request field encoding to use.
...@@ -110,9 +111,9 @@ class RequestMethods(object): ...@@ -110,9 +111,9 @@ class RequestMethods(object):
the body. This is useful for request methods like POST, PUT, PATCH, etc. the body. This is useful for request methods like POST, PUT, PATCH, etc.
When ``encode_multipart=True`` (default), then When ``encode_multipart=True`` (default), then
:func:`urllib3.encode_multipart_formdata` is used to encode :meth:`urllib3.filepost.encode_multipart_formdata` is used to encode
the payload with the appropriate content type. Otherwise the payload with the appropriate content type. Otherwise
:func:`urllib.parse.urlencode` is used with the :meth:`urllib.urlencode` is used with the
'application/x-www-form-urlencoded' content type. 'application/x-www-form-urlencoded' content type.
Multipart encoding must be used when posting files, and it's reasonably Multipart encoding must be used when posting files, and it's reasonably
......
...@@ -2,23 +2,23 @@ from __future__ import absolute_import ...@@ -2,23 +2,23 @@ from __future__ import absolute_import
# For backwards compatibility, provide imports that used to be here. # For backwards compatibility, provide imports that used to be here.
from .connection import is_connection_dropped from .connection import is_connection_dropped
from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers from .request import make_headers
from .response import is_fp_closed from .response import is_fp_closed
from .retry import Retry
from .ssl_ import ( from .ssl_ import (
ALPN_PROTOCOLS, SSLContext,
HAS_SNI, HAS_SNI,
IS_PYOPENSSL, IS_PYOPENSSL,
IS_SECURETRANSPORT, IS_SECURETRANSPORT,
PROTOCOL_TLS,
SSLContext,
assert_fingerprint, assert_fingerprint,
resolve_cert_reqs, resolve_cert_reqs,
resolve_ssl_version, resolve_ssl_version,
ssl_wrap_socket, ssl_wrap_socket,
PROTOCOL_TLS,
) )
from .timeout import Timeout, current_time from .timeout import current_time, Timeout
from .url import Url, get_host, parse_url, split_first
from .retry import Retry
from .url import get_host, parse_url, split_first, Url
from .wait import wait_for_read, wait_for_write from .wait import wait_for_read, wait_for_write
__all__ = ( __all__ = (
...@@ -27,7 +27,6 @@ __all__ = ( ...@@ -27,7 +27,6 @@ __all__ = (
"IS_SECURETRANSPORT", "IS_SECURETRANSPORT",
"SSLContext", "SSLContext",
"PROTOCOL_TLS", "PROTOCOL_TLS",
"ALPN_PROTOCOLS",
"Retry", "Retry",
"Timeout", "Timeout",
"Url", "Url",
...@@ -44,6 +43,4 @@ __all__ = ( ...@@ -44,6 +43,4 @@ __all__ = (
"ssl_wrap_socket", "ssl_wrap_socket",
"wait_for_read", "wait_for_read",
"wait_for_write", "wait_for_write",
"SKIP_HEADER",
"SKIPPABLE_HEADERS",
) )
from __future__ import absolute_import from __future__ import absolute_import
import socket import socket
from urllib3.exceptions import LocationParseError
from ..contrib import _appengine_environ
from ..packages import six
from .wait import NoWayToWaitForSocketError, wait_for_read from .wait import NoWayToWaitForSocketError, wait_for_read
from ..contrib import _appengine_environ
def is_connection_dropped(conn): # Platform-specific def is_connection_dropped(conn): # Platform-specific
...@@ -14,7 +9,7 @@ def is_connection_dropped(conn): # Platform-specific ...@@ -14,7 +9,7 @@ def is_connection_dropped(conn): # Platform-specific
Returns True if the connection is dropped and should be closed. Returns True if the connection is dropped and should be closed.
:param conn: :param conn:
:class:`http.client.HTTPConnection` object. :class:`httplib.HTTPConnection` object.
Note: For platforms like AppEngine, this will always return ``False`` to Note: For platforms like AppEngine, this will always return ``False`` to
let the platform handle connection recycling transparently for us. let the platform handle connection recycling transparently for us.
...@@ -47,7 +42,7 @@ def create_connection( ...@@ -47,7 +42,7 @@ def create_connection(
port)``) and return the socket object. Passing the optional port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance *timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`socket.getdefaulttimeout` global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port) is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection. for the socket to bind as a source address before making the connection.
An host of '' or port 0 tells the OS to use the default. An host of '' or port 0 tells the OS to use the default.
...@@ -63,13 +58,6 @@ def create_connection( ...@@ -63,13 +58,6 @@ def create_connection(
# The original create_connection function always returns all records. # The original create_connection function always returns all records.
family = allowed_gai_family() family = allowed_gai_family()
try:
host.encode("idna")
except UnicodeError:
return six.raise_from(
LocationParseError(u"'%s', label empty or too long" % host), None
)
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res af, socktype, proto, canonname, sa = res
sock = None sock = None
......
This diff is collapsed.
import collections import collections
from ..packages import six from ..packages import six
from ..packages.six.moves import queue from ..packages.six.moves import queue
......
from __future__ import absolute_import from __future__ import absolute_import
from base64 import b64encode from base64 import b64encode
from ..exceptions import UnrewindableBodyError
from ..packages.six import b, integer_types from ..packages.six import b, integer_types
from ..exceptions import UnrewindableBodyError
# Pass as a value within ``headers`` to skip
# emitting some HTTP headers that are added automatically.
# The only headers that are supported are ``Accept-Encoding``,
# ``Host``, and ``User-Agent``.
SKIP_HEADER = "@@@SKIP_HEADER@@@"
SKIPPABLE_HEADERS = frozenset(["accept-encoding", "host", "user-agent"])
ACCEPT_ENCODING = "gzip,deflate" ACCEPT_ENCODING = "gzip,deflate"
try: try:
......
from __future__ import absolute_import from __future__ import absolute_import
import re import re
from collections import namedtuple from collections import namedtuple
from ..exceptions import LocationParseError from ..exceptions import LocationParseError
from ..packages import six from ..packages import six
url_attrs = ["scheme", "auth", "host", "port", "path", "query", "fragment"] url_attrs = ["scheme", "auth", "host", "port", "path", "query", "fragment"]
# We only want to normalize urls with an HTTP(S) scheme. # We only want to normalize urls with an HTTP(S) scheme.
......
import errno import errno
from functools import partial
import select import select
import sys import sys
from functools import partial
try: try:
from time import monotonic from time import monotonic
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment