October 28, 2012

Зачем в python with

Долгое время при работе с файлами из python я писал примерно следующий код:

Без подсветки синтаксиса
def some_func(fname):
    fd = open(fname)
    some_data_processing(fd.read())
    return result

Тут предполагается, что в любом случае при выходе из функции переменная fd уничтожится и вместе с ней закроется файл и все будут жить долго и счастливо.

Но что будет если в some_data_processing произойдет исключение?

Например так:

Без подсветки синтаксиса
import sys

class TestClass(object):
    def __del__(self):
        print "I'm deleted"

def data_process():
    obj = TestClass()
    raise IndexError()

try:
    data_process()
except:
    print "In exception handler"

print "after except"

На консоли появляется:

    In exception handler
    after except
    I'm deleted

Почему-то "In exception handler" и "after except" выводятся раньше "I'm deleted".

Первая проблема в том, что вместе с исключением питон хранит и трейс стека, содержащий все фреймы вплоть до породившего исключение. А внутри фрейма живет f_locals - словарь локальных переменных, и именно он имеет ссылку на экземпляр класса TestClass. Таким образом до окончания обработки исключения obj будет жить точно. Почему же "after except" появляется раньше чем "I'm deleted"? Было бы логично чистить трейс после успешного выхода из блока try. Дело в том что 2.X питон не всегда чистит внутренние структуры после обработки исключения и в общем случае вы должны явно вызывать функцию sys.exc_clear чтобы очистить их. Когда я подошел с этим вопросом к Larry Hastings (одному из основных разработчиков ядра питона) ему потребовалось около 20ти минут, что-бы понять что происходит и найти в документации sys.exc_clear. (Правда стоит отметить, что он давно использует 3.X, где это поведение стало адекватнее.) В 3.X это поведение улучшили, и теперь sys.exc_clear автоматически вызывается окончанию обработки исключения.

Кстати, если вы напишете примерно такой код:

Без подсветки синтаксиса
try:
    data_process()
except:
    fr = sys.exc_info()[2]
    del fr

то не забудьте удалить fr используя del, как в последней строке - иначе он образует циклическую ссылку с текущим фреймом и тогда все станет совсем плохо.

Стоит отметить, что подобное поведение проявляется не всегда. Например следующий код исполняется более предсказуемо:

Без подсветки синтаксиса
import sys

class TestClass(object):
    def __del__(self):
        print "I'm deleted"

def data_process():
    fd = TestClass()
    try:
        raise IndexError()
    except:
        print "In internal exception handler"

data_process()
print "after except"
    In internal exception handler
    I'm deleted
    after except

В общем что-бы гарантированно избавить себя от этих проблем нужно явно закрывать все файлы и прочие объекты или так:

Без подсветки синтаксиса
fd = open(fname)
try:
    process_code()
finally:
    fd.close()

или так:

Без подсветки синтаксиса
with open(fname) as fd:
    process_code()

собственное with именно для этого и был сделан. Без его использования вы рискуете исчерпать лимит на дескрипторы или что-там-еще в зависимости от объектов. Впрочем это только начало печальной истории, продолжение дальше.

Ссылки:
          docs.python.org/2/library/sys.html#sys.exc_clear
          nuitka.net

Исходники этого и других постов со скриптами лежат тут - github.com/koder-ua. При использовании их, пожалуйста, ссылайтесь на koder-ua.blogspot.com.

44 comments:

Goran said...

Хм... попробовал под Ubuntu 12.10 (там как раз оба Python 2.x и 3.x из коробки) - результат один и тот же

igor@ubuntu:~/Dev/PythonResearch$ python except.py
In internal exception handler
I'm deleted
after except
igor@ubuntu:~/Dev/PythonResearch$ python3 except.py
In internal exception handler
I'm deleted
after except

версия 2.x Python 2.7.3 - либо очень произвольно зависит от сборщика мусора (от его выполнения), либо в самой последней версии 2.7.3 пофиксили до нормального поведения :) Вообще интересный момент

Unknown said...

В случае если исключение отлавливается внутри функциимне не удалось воспроизвести это поведение.
В случае же, если исключение отлавливается на уровне модуля(первый пример) - востпроизводится на 100% на 2.7

Unknown said...

Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

nivatha said...

Nice post. By reading your blog, i get inspired and this provides some useful information. Thank you for posting this exclusive post for our vision. 
Data Science training in marathahalli
Data Science training in btm
Data Science training in rajaji nagar
Data Science training in chennai
Data Science training in kalyan nagar
Data Science training in electronic city
Data Science training in USA
Data science training in pune

Mounika said...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
python training in pune
python online training
python training in OMR

shethal said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
Devops training in tambaram
DevOps online Training
DevOps Training in USA

ummayasri said...

This is such a great post, and was thinking much the same myself. Another great update.
Blue Prism Training Course in Pune

Blue Prism Training Institute in Bangalore

MOUNIKA said...

Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.
Splunk Training From India

Dellboomi Training From India

pragyachitra said...

Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…


angularjs Training in chennai
angularjs-Training in pune

angularjs-Training in chennai

angularjs Training in chennai

angularjs-Training in tambaram

john brito said...

I’m planning to start my blog soon, but I’m a little lost on everything. Would you suggest starting with a free platform like Word Press or go for a paid option? There are so many choices out there that I’m completely confused. Any suggestions? Thanks a lot.
Best AWS Training in Marathahalli | AWS Training in Marathahalli
Amazon Web Services Training in Anna Nagar, Chennai |Best AWS Training in Anna Nagar, Chennai
AWS Training in Velachery | Best AWS Course in Velachery,Chennai
Best AWS Training in Chennai | AWS Training Institutes |Chennai,Velachery

Riya Raj said...

Really great blog... Thanks for your information
Selenium Training in Bangalore
Selenium Training in Coimbatore
Selenium Course in Bangalore
selenium course in coimbatore
Java Training in Bangalore
Python Training in Bangalore
IELTS Coaching in Madurai
IELTS Coaching in Coimbatore

Java Training in Coimbatore

suresh said...

Excellent Article. Thanks Admin

Data Science Training in Chennai

DevOps Training in Chennai

Hadoop Big Data Training

Python Training in Chennai

raybon said...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.

devops online training

aws online training

data science with python online training

data science online training

rpa online training

rose said...

Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
Microsoft Azure online training
Selenium online training
Java online training
Python online training
uipath online training

anusha said...


Selenium Training in Chennai Searching for Best Selenium Training in Chennai ?.BITA Academy is the Best Selenium Training Institute in Chennai.. Our Selenium testing course is taught by Experts.Become a Selenium Certified Professional.

creative web solution said...

Are you looking for a maid for your home to care your baby,patient care taker, cook service or a japa maid for your pregnent wife we are allso providing maid to take care of your old parents.we are the best and cheapest service provider in delhi for more info visit our site and get all info.

maid service provider in South Delhi
maid service provider in Dwarka
maid service provider in Gurgaon
maid service provider in Paschim Vihar
cook service provider in Paschim Vihar
cook service provider in Dwarka
cook service provider in south Delhi
baby care service provider in Delhi NCR
baby care service provider in Gurgaon
baby care service provider in Dwarka
baby service provider in south Delhi
servant service provider in Delhi NCR
servant service provider in Paschim Vihar
servant Service provider in South Delhi
japa maid service in Paschim Vihar
japa maid service in Delhi NCR
japa maid service in Dwarka
japa maid service in south Delhi
patient care service in Paschim Vihar
patient care service in Delhi NCR
patient care service in Dwarka
Patient care service in south Delhi

creative web solution said...

Alleyaaircool is the one of the best home appliances repair canter in all over Delhi we deals in repairing window ac, Split ac , fridge , microwave, washing machine, water cooler, RO and more other home appliances in cheap rates

Window AC Repair in vaishali
Split AC Repair in indirapuram
Fridge Repair in kaushambi
Microwave Repair in patparganj
Washing Machine Repair in vasundhara
Water Cooler Repair in indirapuram
RO Service AMC in vasundhara
Any Cooling System in vaishali
Window AC Repair in indirapuram

Saketh said...

Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading Python training in pune new articles. Keep up the good work!

easylearn said...

Hi,
Best article, very useful and well explanation. Your post is extremely incredible.Good job & thank you very much for the new information, i learned something new. Very well written. It was sooo good to read and usefull to improve knowledge. Who want to learn this information most helpful. One who wanted to learn this technology IT employees will always suggest you take Python Training Institutes In Btm.

Vijiaajith said...

nice
freeinplanttrainingcourseforECEstudents
internship-in-chennai-for-bsc
inplant-training-for-automobile-engineering-students
freeinplanttrainingfor-ECEstudents-in-chennai
internship-for-cse-students-in-bsnl
application-for-industrial-training

Vijiaajith said...

nice
interview-questions/aptitude/permutation-and-combination/how-many-groups-of-6-persons-can-be-formed

tutorials/oracle/oracle-delete

technology/chrome-flags-complete-guide-enhance-browsing-experience/

interview-questions/aptitude/time-and-work/a-alone-can-do-1-4-of-the-work-in-2-days


interview-questions/programming/recursion-and-iteration/integer-a-40-b-35-c-20-d-10-comment-about-the-output-of-the-following-two-statements

Padminiprwatech said...

Thanks a lot for giving great kind of information. So useful and practical for me. Excellent blog and very informative, nice work keep updating. If you are looking for any Python Programming language related information, check our Python training institute in bangalore web page. Thanks a lot.

raju said...

nice...!
brunei darussalam hosting
inplant training in chennai

raju said...

nice post...!
dominican republic web hosting
iran hosting
palestinian territory web hosting
panama web hosting
syria hosting
services hosting
afghanistan shared web hosting
andorra web hosting
belarus web hosting

shiv said...

nice....................
vietnam web hosting
google cloud server hosting
canada telus cloud hosting
algeeria hosting
angola hostig
shared hosting
bangladesh hosting
botswana hosting
central african republi hosting
shared hosting

shree said...

nice...
luxembourg web hosting
mauritius web hosting mongolia web hosting
namibia web hosting
norway web hosting
rwanda web hosting
spain hosting
turkey web hosting
venezuela hosting
vietnam shared web hosting

preethi minion said...

nice..
afghanistan hosting
angola hosting
afghanistan web hosting
bahrain web hosting
belize web hosting
india shared web hosting

preethi minion said...

italy web hosting
suden web hosting
tunisia hosting
uruguay web hosting
inplant training in chennai

shri said...

very nice.....
internship in chennai for ece students
internships in chennai for cse students 2019
Inplant training in chennai
internship for eee students
free internship in chennai
eee internship in chennai
internship for ece students in chennai
inplant training in bangalore for cse
inplant training in bangalore
ccna training in chennai


KUTTYMA said...

nice information....

winter internship for engineering students
internship for mca students
inplant training for eee students
inplant training for eee students
java training in chennai
internships for eee students in hyderabad
ece internship
internship certificate for mechanical engineering students
internship in nagpur for computer engineering students
kaashiv infotech internship
internship for aeronautical engineering students in india 2019

sibiselvan said...

it is excellent blogs...!!
inplant training for diploma students
mechanical internship in chennai
civil engineering internship in chennai
internship for b.arch students in chennai
internship for ece students in core companies in chennai
internship in chandigarh for ece
industrial training report for computer science engineering on python
internship for automobile engineering students in chennai
big data training in chennai
ethical hacking internship in chennai

anuarun said...

nice information......
ree internship in bangalore for computer science students
internship for aeronautical engineering
internship for eee students in hyderabad
internship in pune for computer engineering students 2018
kaashiv infotech internship fees
industrial training certificate format for mechanical engineering students
internship report on machine learning with python
internship for biomedical engineering students in chennai
internships in bangalore for cse
internship in coimbatore for ece

shree said...

excellent
kyrgyzstan web hosting
lebanon web hosting
lithuania web hosting
macao hosting
madagascar web hosting
pakistan hosting
panama shared web hosting
paraguay web hosting
peru web hosting
philippines hosting

raju said...

very nice blogger thanks for sharing............!!!
poland web hosting
russian federation web hosting
slovakia web hosting
spain web hosting
suriname
syria web hosting
united kingdom
united kingdom shared web hosting
zambia web hosting

preethi minion said...

Awesome blogs.....
robotics courses
inplant training in chennai for eee students
paid internships in hyderabad for cse students
list of architectural firms for internship in india
internship for mca students
matlab training in chennai
final year project for it
internship for production engineering students
aeronautical internship
inplant training report for civil engineering

sibiselvan said...


it is best blogs for you...!!!
paid internships in pune for computer science students
machine learning training in chennai
data science internship in chennai
dot net training in chennai
kaashiv infotech chennai
internship for aeronautical engineering students in india
internship in automobile industry
big data internship in chennai
machine learning internship in chennai
internship in chennai for it students

KUTTYMA said...

IT IS A BEST ONE FOR SEARCHING....
kaashiv infotech internship in bangalore
internship for ece
mba internship
final year project proposal for information technology
internships in chennai for ece students
companies for industrial visit in chennai for cse students
internship in bangalore for eee
internship in chennai
inplant training certificate format for mechanical engineering
internship for aeronautical engineering students in chennai


shri said...

very good...
internship report on python
free internship in chennai for ece students
free internship for bca
internship for computer science engineering students in india
internships in hyderabad for cse students 2018
electrical companies in hyderabad for internship
internships in chennai for cse students 2019
internships for ece students
inplant training in tcs chennai
internship at chennai

anuarun said...

Good information
kaashiv infotech internship fees
internship
internship in chennai for cse students
internships in hyderabad for eee students 2019
inplant training for cse
internship for ece students in coimbatore
internship for cse
internship for cse 3rd year students
inplant training in chennai for ece students
summer training for 3rd year electronics and communication engineering students

shalini said...


good.....
kaashiv infotech pune
industrial training report for electronics and communication
internships for cse
internship for automobile engineering students in bangalore
internships in bangalore for eee students
internship for civil engineering students in chennai 2019
internship in automobile companies in chennai
robotics chennai
final year projects for information technology

nivetha said...

hiii nyc..
internships for cse students in bangalore
internship for cse students
industrial training for diploma eee students
internship in chennai for it students
kaashiv infotech in chennai
internship in trichy for ece
inplant training for ece
inplant training in coimbatore for ece
industrial training certificate format for electrical engineering students
internship certificate for mechanical engineering students

karthickannan said...

nice...
coronavirus update
inplant training in chennai
inplant training
inplant training in chennai for cse
inplant training in chennai for ece
inplant training in chennai for eee
inplant training in chennai for mechanical
internship in chennai
online internship

divya said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.thank you si much.
Ai & Artificial Intelligence Course in Chennai
PHP Training in Chennai
Ethical Hacking Course in Chennai Blue Prism Training in Chennai
UiPath Training in Chennai

Rashika said...

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work.
AWS training in chennai | AWS training in annanagar | AWS training in omr | AWS training in porur | AWS training in tambaram | AWS training in velachery