PyFITS: hdulist.writeto()

I'm extracting extensions from a multi-extension FITS file, manipulate the data, and save the data (with the extension's header information) to a new FITS file.

To my knowledge pyfits.writeto() does the task. However, when I give it a data parameter in the form of an array, it gives me the error:

    'AttributeError: 'numpy.ndarray' object has no attribute 'lower''

Here is a sample of my code:

    'file = 'hst_11166_54_wfc3_ir_f110w_drz.fits'
     hdulist = pyfits.open(dir + file)'
     sci = hdulist[1].data # science image data
     exp = hdulist[5].data # exposure time data
     sci = sci*exp # converts electrons/second to electrons
     file = 'test_counts.fits'

     hdulist.writeto(file,sci,clobber=True)

     hdulist.close()

I appreciate any help with this. Thanks in advance.

via Chebli Mohamed

Form for user registration in Django

I`m trying to write the register form for my website, my code is as follows. the problem is when I press the submit button in the registration page nothing actually happens... I already appreciate your help.

register.html

<form id="reg_form" action="/register/" method="post" >
    <div class="container">
        <div class="row">

            <div class="col-md-6">
                <div class="corner" style="margin-top: 5pt;">
                    <label>قوانین عضویت:</label>

                    <div class="form-group">
                        <label for="accept">با قوانین سایت موافقم</label>
                        <input type="checkbox"  id="accept"/>
                    </div>
                </div>

               <button type="submit" class="btn btn-success" data-dismiss="modal" style="width: 100%;font-size: xx-large; margin-top: 10pt;">ثبت</button>
            </div>
            <div class="col-md-6">
                <div class="form-group">
                    <label for="fname">نام:</label>
                    <input type="text" class="form-control" id="fname"/>
                </div>
                <div class="form-group">
                    <label for="lname">نام خانوادگی:</label>
                    <input type="text" class="form-control" id="lname">
                </div>
                <div class="form-group">
                    <label for="email">ایمیل:</label>
                    <input type="email" class="form-control" id="email">
                </div>
                <div class="form-group">
                    <label for="phone">شماره تماس:</label>
                    <input type="text" class="form-control" id="phone">
                </div>
                <div class="form-group">
                    <label for="address">آدرس:</label>
                    <input type="text" class="form-control" id="address">
                </div>
                <div class="form-group">
                    <label for="pwd">رمز عبور:</label>
                    <input type="password" class="form-control" id="pwd">
                </div>
                <div class="form-group">
                    <label for="pwd2">تکرار رمز عبور:</label>
                    <input type="password" class="form-control" id="pwd2">
                </div>

            </div>
        </div>

    </div>
    </form>

views.py ________________________________________________________________

def register(request):
    if request.method == 'POST':
        fname=request.POST['fname']
        lname=request.POST['lname']
        email=request.POST['email']
        phone=request.POST['phone']
        address=request.POST['address']
        pwd=request.POST['pwd']

        user=User.objects.create_user(username=email, password=pwd )
        user.save()
        return HttpResponseRedirect("/home/")
    return render(request, 'Register.html')

via Chebli Mohamed

pip install requests[security] vs pip install requests: Difference

I am using Ubuntu 14.04 with python version 2.7.6. Today, when I created a new virtualenv and tried doing pip install requests , I got the error InsecurePlatformWarning.

I resolved this issue by following the instructions over here

SSL InsecurePlatform error when using Requests package

But I want to understand what is the actual difference between these two commands: pip install requests[security] and pip install requests .

1) Why does the former install 3 additional packages?

2) Are there any things that I need to take care about when I push the code to production?

3) Do they both behave the same generally?

I searched a lot on stackoverflow and elsewhere, but couldn't find the answer. If it has already been answered then please post the link.

Thanks.

via Chebli Mohamed

Plot 4th dimension with Python

I would to know if there is the possibility to plot in four dimensions using python. In particular I would to have a tridimensional mesh X, Y, Z and f(X,Y,Z) = 1 or f(X,Y,Z) = 0. So I need to a symbol (for example "o" or "x") for some specific point (X,Y,Z). I don't need to a color scale.

Note that I have 100 matrices (512*512) composed by 1 or 0: so my mesh should be 512*512*100.

I hope I have been clear! Thanks.

EDIT: This is my code:

X = np.arange(W.shape[2])
Y = np.arange(W.shape[1])
Z = np.arange(W.shape[0])
X, Y, Z = np.meshgrid(X, Y, Z)
fig = plt.figure()
ax = fig.gca(projection='3d')
for z in range(W.shape[0]):
    indexes = np.where(W[z])
    ax.scatter(X[indexes], Y[indexes], ???, marker='.')

ax.set_xlabel('X = columns')
ax.set_ylabel('Y = rows')
ax.set_zlabel('Z')
plt.show()

W is my tridimensional matrix, so: W[0], W[1], etc are 512x512 matrices. My question is: what have I to write insted of ??? in my code. I know I shouldn't ask this, but I can't understand the idea.

via Chebli Mohamed

Extract part of a url using pattern matching in python

I want to extract part of a url using pattern matching in python from a list of links

Examples:

http://ift.tt/1KNZ7nE  
http://ift.tt/1g4Fr1i   

This is my regex:

re.match(r'(http?|ftp)(://[a-zA-Z0-9+&/@#%?=~_|!:,.;]*)(.\b[a-z]{1,3}\b)(/about[a-zA-Z-_]*/?)', str(href), re.IGNORECASE)  

I want to get links ending only with /about or /about/
but the about regex selects all links with "about" word in it

via Chebli Mohamed

python ZeroDivisionError: float division by zero - how to treat it as an exception

I need to calculate the percentage of winning rate and I am having trouble with ZeroDivisionError: float division by zero when there is zero in the denominator. I would like treat it as an exception, print zero and move on to next array. Any assistance would be appreciated.

YearWinLose1 = '0'
YearWinLoseTotalHard2 = '0'

YearWinLoseTotalHard2 = float(YearWinLose1) + float(YearWinLose1)
YearWinLosePercent3 = float(YearWinLose1)/float(YearWinLoseTotalHard2)*100

if YearWinLoseTotalHard2 == 0:
   YearWinLosePercent3 == 0
   print YearWinLosePercent3
else: 
   print YearWinLosePercent3

via Chebli Mohamed

pandas - splitting columns

I have some data imported from a csv, to create something similar I used this:

data = pd.DataFrame([[1,0,2,3,4,5],[0,1,2,3,4,5],[1,1,2,3,4,5],[0,0,2,3,4,5]], columns=['split','sex', 'group0Low', 'group0High', 'group1Low', 'group1High'])
means = data.groupby(['split','sex']).mean()

so the dataframe looks something like this:

        group0Low   group0High  group1Low   group1High
split   sex             
0   0   123 54  95  265
    1   98  111 120 220
1   0   211 300 150 190
    1   132 250 139 86

You'll notice that each column actually contains 2 variables (group# and height). It was set up this way for running repeated measures anova in SPSS, but that's beyond the point.

Ideally, I want to use pandas to split the columns up, so I can also groupby "group", like this (I actually screwed up the order of the numbers, but hopefully the idea is clear):

            low high
split   sex group       
    0   0   95  265
0   0   1   123 54
    1   0   120 220
    1   1   98  111
1   0   0   150 190
    0   1   211 300
    1   0   139 86
    1   1   132 250

is how do I achieve this in pandas? Is it even possible?

via Chebli Mohamed

Python xlrd read DateTime as serial number

I have an excel which is a DateTime format. But when I read the sheet it reads as serial number.

For example, in the sheet it's 08/03/2015 but xlrd reads as 42219.

The code is pretty simple

workbook = xlrd.open_workbook(file_contents=excel_contents)
the_sheet = workbook.sheet_names()[0]
number_of_rows = the_sheet.nrows - 1
curr_row = 0
data = []

while curr_row < number_of_rows:
    curr_row += 1
    air_date = the_sheet.cell_value(curr_row, 1)
    air_time = the_sheet.cell_value(curr_row, 2)
    schedule_time = '{}:{}'.format(air_date, air_time)
    unaware = datetime.strptime(schedule_time, '%m/%d/%Y:%I:%M %p') 

The air_date which I use to translate to datetime in Python bombs out because the format is different.

This is the example excel.

And here's the error I got

builtins.ValueError ValueError: time data '42219.0:06:00 AM' does not match format '%m/%d/%Y:%I:%M %p'

via Chebli Mohamed

GridFS on App Engine?

I get a server error when I try to import gridfs in my app engine application made with Flask.

But, when I run that app locally, it runs fine. How can I import gridfs on App Engine?

via Chebli Mohamed

Python xlrd Exception: Unknown cell type 'n+'

I have a python script that downloads an xlsx file and parses the contents, however it's started failing at the line

 workbook = xlrd.open_workbook(fname)

with the error "Exception: Unknown cell type 'n+' in rowx=1 colx=4"

Can anyone point me in the right direction with regards to what the n+ cell type is or how to move past this error

Alternatively if anyone knows of a python module that will read an xlsx file without tripping over this cell value.

Cheers

-- EDIT --

After unzipping the file I found this in the xml of the worksheet

<c r="E2" t="n+"><v>18</v></c>

via Chebli Mohamed

Python - Random Forest - Iteratively adding trees

Good afternoon! Sorry for my English, but I need your help. I am doing some machine learning task on Python. I need to build RandomForest and than build a graph that will show how the quality of the training and test samples depends on the number of trees in the Random Forest. It is necessary each time to build a new Random Forest with a certain number of trees? Or i can somehow iteratively add trees (if it possible, can you give the exapmle of code how to do that)?

via Chebli Mohamed

Matplotlib plotting- two data sets with different no. of data points

I am trying to plot two data sets into the same graph using matplotlib. No. of data points for the first pair of (x1,y1) is 150. No. of data points for the second pair (x2,y2) is 100. Please note that the range for x1 and x2 is from 0.0 to 1.0. Only the no. of data points are different. When I try to plot these into the same graph for comparison, the error message appears that "x and y must have same first dimension". Code snippet below for your reference:

def main():
 x1,y1 = np.loadtxt('txt1.txt',unpack=True)
 x2,y2 = np.loadtxt('txt2.txt',unpack=True)
 plt.figure()
 plt.plot(x1,y1)
 plt.plot(x2,y2)
main()

via Chebli Mohamed

Django pass render_to_response template in other template

this is probably a question for absolute beginners since i'm fairly new to progrmaming. I've searched for couple of hours for an adequate solution, i don't know what else to do.

Following problem. I want to have a view that displays. e.g. the 5 latest entries & 5 newest to my database (just an example)

#views.py
import core.models as coremodels

class LandingView(TemplateView):
    template_name = "base/index.html"

    def index_filtered(request):
        last_ones = coremodels.Startup.objects.all().order_by('-id')[:5]
        first_ones = coremodels.Startup.objects.all().order_by('id')[:5]
        return render_to_response("base/index.html", 
        {'last_ones': last_ones,   'first_ones' : first_ones})  

Index.html shows the HTML content but not the content of the loop

#index.html

<div class="col-md-6">
    <p> Chosen Items negative:</p>
    {% for startup in last_ones %}
        <li><p>{{ startup.title }}</p></li>
    {% endfor %}
</div>

<div class="col-md-6">
    <p> Chosen Items positive:</p>
    {% for startup in first_ones %}
       <li><p>{{ startup.title }}</p></li>
    {% endfor %}

Here my problem:

How can I get the for loop to render the specific content?

I think Django show render_to_response in template comes very close to my problem, but i don't see a valid solution there.

Thank you for your help.

Chris

-- I edited my code and problem description based on the solutions provided in this thread

via Chebli Mohamed

how to repeat same operations for dataset subset

I have a dataset of transactions. What I'm doing is determining what customers have done just one purchase and what customer have purchased more than once.

what I did is:

 df1['Counts'] = df1.groupby(['email_address']).transform('count') # get count of repeating emails/customers

 result.ix[result.Counts >= 2,'Repeat'] = 1 # create new column with 1 for those that repeated and 0 for those who didn't
 result.ix[result.Counts <= 1,'Repeat'] = 0

This gives me the overall count. However, the data frame has an additional column with the name of the website in which the purchase was made. What I want to do is to find customers that bought more than once in the same website.

Any idea how to change the previous function to get result as a column value breakdown?

via Chebli Mohamed

Error On while accessing tweepy on google app engine

I am trying to get user timeline using tweepy on google app engine.

I was facing issue of ssl on local machine that I have resolved using "ImportError: No module named _ssl" with dev_appserver.py from Google App Engine

but when i deployed it on google app engine it is not working.

here is my screenshot

what I am missing?

via Chebli Mohamed

Tell if text of PDF is visible or not

I'm parsing some PDF files using the pdfminer library.

I need to know if the document is a scanned document, where the scanning machine places the scanned image on top and OCR-extracted text in the background.

Is there a way to identify if text is visible, as OCR machines do place it on the page for selection.

Generally the problem is distinguishing between two very different, but similar looking cases.

In one case there's an image of a scanned document that covers most of the page, with the OCR text behind it.

Here's the PDF as text with the image truncated: http://ift.tt/1ONm1cL

In the other case there's a background image that covers most of the page with the text in front of it.

Telling them apart is proving difficult for me.

via Chebli Mohamed

Python How to copy files inside a zip to another zip in memory?

Purpose

Split a zip archive into smaller zip archives with an evenly distributed # of files per new zip.

Example

source zip (100 files)

• src/100-Test.zip

destination zips (25 files each):

• destination/1.zip
• destination/2.zip
• destination/3.zip
• destination/4.zip

Description

So I have been able to open the zip file and iterate through the contents to split them up, but I have not been able to write to the file. Since I didn't do anything with the zip contents I didn't think I had to do any StringIO stuff or anything?

Code

zipFileNameSrc = '100-Test.zip'
zipFile = open(zipFileNameSrc)
unzippedFile = zipfile.ZipFile(zipFile)
imgList = [(s, unzippedFile.read(s)) for s in unzippedFile.namelist() if (".jpg" or ".JPG") in s]
#image names: imgList[i][0]  and  images: imgList[i][1]

#...
#...additional logic to split into sets of 25 images
#...fileTuplesList = imgList[:25]
zipNo = 1
#zipFileDest = destination + "/" + zipSrcNm + "/" + zipNo.__str__() + ".zip"
zipFileName = zipNo.__str__() + ".zip"
zipOut = zipfile.ZipFile(zipFileName, 'w')
for i in xrange(len(fileTuplesList)):
    fileNameAndPath = fileTuplesList[i][0]
    actualFile = fileTuplesList[i][1]
    zipOut.write(fileNameAndPath, actualFile)
zipOut.close()
#move_files(zipFileName, zipFileDest)

Error

I get on this on line zipOut.write(fileNameAndPath, actualFile)

OSError: [Errno 2] No such file or directory: '100-Test/17.jpg'

Bonus

How to save the zip file to a different folder than where my script is?

via Chebli Mohamed

Celery error : result.get times out

I've installed Celery and I'm trying to test it with the Celery First Steps Doc.

I tried using both Redis and RabbitMQ as brokers and backends, but I can't get the result with :

result.get(timeout = 10)

Each time, I get this error :

Traceback (most recent call last): File "", line 11, in File "/home/mehdi/.virtualenvs/python3/lib/python3.4/site-packages/celery/result.py", line 169, in get no_ack=no_ack, File "/home/mehdi/.virtualenvs/python3/lib/python3.4/site-packages/celery/backends/base.py", line 225, in wait_for raise TimeoutError('The operation timed out.') celery.exceptions.TimeoutError: The operation timed out.

The broker part seems to work just fine : when I run this code

from celery import Celery

app = Celery('tasks', backend='redis://localhost/', broker='amqp://')

@app.task
def add(x, y):
    return x + y

result = add.delay(4,4)

I get (as expected)

[2015-08-04 12:05:44,910: INFO/MainProcess] Received task: tasks.add[741160b8-cb7b-4e63-93c3-f5e43f8f8a02]

[2015-08-04 12:05:44,911: INFO/MainProcess] Task tasks.add[741160b8-cb7b-4e63-93c3-f5e43f8f8a02] succeeded in 0.0004287530000510742s: 8

P.S : I'm using Xubuntu 64bit

EDIT :

My app.conf

{'CELERY_RESULT_DB_TABLENAMES': None, 'BROKER_TRANSPORT_OPTIONS': {}, 'BROKER_USE_SSL': False, 'CELERY_BROADCAST_QUEUE': 'celeryctl', 'EMAIL_USE_TLS': False, 'CELERY_STORE_ERRORS_EVEN_IF_IGNORED': False, 'CELERY_CREATE_MISSING_QUEUES': True, 'CELERY_DEFAULT_QUEUE': 'celery', 'CELERY_SEND_TASK_SENT_EVENT': False, 'CELERYD_TASK_TIME_LIMIT': None, 'BROKER_URL': 'amqp://', 'CELERY_EVENT_QUEUE_EXPIRES': None, 'CELERY_DEFAULT_EXCHANGE_TYPE': 'direct', 'CELERYBEAT_SCHEDULER': 'celery.beat:PersistentScheduler', 'CELERY_MAX_CACHED_RESULTS': 100, 'CELERY_RESULT_PERSISTENT': None, 'CELERYD_POOL': 'prefork', 'CELERYD_AGENT': None, 'EMAIL_HOST': 'localhost', 'CELERY_CACHE_BACKEND_OPTIONS': {}, 'BROKER_HEARTBEAT': None, 'CELERY_RESULT_ENGINE_OPTIONS': None, 'CELERY_RESULT_SERIALIZER': 'pickle', 'CELERYBEAT_SCHEDULE_FILENAME': 'celerybeat-schedule', 'CELERY_REDIRECT_STDOUTS_LEVEL': 'WARNING', 'CELERY_IMPORTS': (), 'SERVER_EMAIL': 'celery@localhost', 'CELERYD_TASK_LOG_FORMAT': '[%(asctime)s: %(levelname)s/%(processName)s] %(task_name)s[%(task_id)s]: %(message)s', 'CELERY_SECURITY_CERTIFICATE': None, 'CELERYD_LOG_COLOR': None, 'CELERY_RESULT_EXCHANGE': 'celeryresults', 'CELERY_TRACK_STARTED': False, 'CELERY_REDIS_PASSWORD': None, 'BROKER_USER': None, 'CELERY_COUCHBASE_BACKEND_SETTINGS': None, 'CELERY_RESULT_EXCHANGE_TYPE': 'direct', 'CELERY_REDIS_DB': None, 'CELERYD_TIMER_PRECISION': 1.0, 'CELERY_REDIS_PORT': None, 'BROKER_TRANSPORT': None, 'CELERYMON_LOG_FILE': None, 'CELERYD_CONCURRENCY': 0, 'CELERYD_HIJACK_ROOT_LOGGER': True, 'BROKER_VHOST': None, 'CELERY_DEFAULT_EXCHANGE': 'celery', 'CELERY_DEFAULT_ROUTING_KEY': 'celery', 'CELERY_ALWAYS_EAGER': False, 'EMAIL_TIMEOUT': 2, 'CELERYD_TASK_SOFT_TIME_LIMIT': None, 'CELERY_WORKER_DIRECT': False, 'CELERY_REDIS_HOST': None, 'CELERY_QUEUE_HA_POLICY': None, 'BROKER_PORT': None, 'CELERYD_AUTORELOADER': 'celery.worker.autoreload:Autoreloader', 'BROKER_CONNECTION_TIMEOUT': 4, 'CELERY_ENABLE_REMOTE_CONTROL': True, 'CELERY_RESULT_DB_SHORT_LIVED_SESSIONS': False, 'CELERY_EVENT_SERIALIZER': 'json', 'CASSANDRA_DETAILED_MODE': False, 'CELERY_REDIS_MAX_CONNECTIONS': None, 'CELERY_CACHE_BACKEND': None, 'CELERYD_PREFETCH_MULTIPLIER': 4, 'BROKER_PASSWORD': None, 'CELERY_BROADCAST_EXCHANGE_TYPE': 'fanout', 'CELERY_EAGER_PROPAGATES_EXCEPTIONS': False, 'CELERY_IGNORE_RESULT': False, 'CASSANDRA_KEYSPACE': None, 'EMAIL_HOST_PASSWORD': None, 'CELERYMON_LOG_LEVEL': 'INFO', 'CELERY_DISABLE_RATE_LIMITS': False, 'CELERY_TASK_PUBLISH_RETRY_POLICY': {'interval_start': 0, 'interval_max': 1, 'max_retries': 3, 'interval_step': 0.2}, 'CELERY_SECURITY_KEY': None, 'CELERY_MONGODB_BACKEND_SETTINGS': None, 'CELERY_DEFAULT_RATE_LIMIT': None, 'CELERYBEAT_SYNC_EVERY': 0, 'CELERY_EVENT_QUEUE_TTL': None, 'CELERYD_POOL_PUTLOCKS': True, 'CELERY_TASK_SERIALIZER': 'pickle', 'CELERYD_WORKER_LOST_WAIT': 10.0, 'CASSANDRA_SERVERS': None, 'CELERYD_POOL_RESTARTS': False, 'CELERY_TASK_PUBLISH_RETRY': True, 'CELERY_ENABLE_UTC': True, 'CELERY_SEND_EVENTS': False, 'BROKER_CONNECTION_MAX_RETRIES': 100, 'CELERYD_LOG_FILE': None, 'CELERYD_FORCE_EXECV': False, 'CELERY_CHORD_PROPAGATES': True, 'CELERYD_AUTOSCALER': 'celery.worker.autoscale:Autoscaler', 'CELERYD_STATE_DB': None, 'CELERY_ROUTES': None, 'CELERYD_TIMER': None, 'ADMINS': (), 'BROKER_HEARTBEAT_CHECKRATE': 3.0, 'CELERY_ACCEPT_CONTENT': ['json', 'pickle', 'msgpack', 'yaml'], 'BROKER_LOGIN_METHOD': None, 'BROKER_CONNECTION_RETRY': True, 'CELERY_TIMEZONE': None, 'CASSANDRA_WRITE_CONSISTENCY': None, 'CELERYBEAT_MAX_LOOP_INTERVAL': 0, 'CELERYD_LOG_LEVEL': 'WARN', 'CELERY_REDIRECT_STDOUTS': True, 'BROKER_POOL_LIMIT': 10, 'CELERY_SECURITY_CERT_STORE': None, 'CELERYD_CONSUMER': 'celery.worker.consumer:Consumer', 'CELERY_INCLUDE': (), 'CELERYD_MAX_TASKS_PER_CHILD': None, 'CELERYD_LOG_FORMAT': '[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'CELERY_ANNOTATIONS': None, 'CELERY_MESSAGE_COMPRESSION': None, 'CASSANDRA_READ_CONSISTENCY': None, 'EMAIL_USE_SSL': False, 'CELERY_SEND_TASK_ERROR_EMAILS': False, 'CELERY_QUEUES': None, 'CELERY_ACKS_LATE': False, 'CELERYMON_LOG_FORMAT': '[%(asctime)s: %(levelname)s] %(message)s', 'CELERY_TASK_RESULT_EXPIRES': datetime.timedelta(1), 'BROKER_HOST': None, 'EMAIL_PORT': 25, 'BROKER_FAILOVER_STRATEGY': None, 'CELERY_RESULT_BACKEND': 'rpc://', 'CELERY_BROADCAST_EXCHANGE': 'celeryctl', 'CELERYBEAT_LOG_FILE': None, 'CELERYBEAT_SCHEDULE': {}, 'CELERY_RESULT_DBURI': None, 'CELERY_DEFAULT_DELIVERY_MODE': 2, 'CELERYBEAT_LOG_LEVEL': 'INFO', 'CASSANDRA_COLUMN_FAMILY': None, 'EMAIL_HOST_USER': None}

via Chebli Mohamed

Python: Combine Multiline to Singleline

I have input file with data:

2015-06-05 16:22:10 Payload-Line-1

Payload-Line-2 

2015-06-05 16:22:11 Payload-Line-1

Payload-Line-2

Payload-Line-3

2015-06-05 16:22:12 Payload-Line-1

I try to make out file like:

2015-06-05 16:22:10 Payload-Line-1
Payload-Line-2 

2015-06-05 16:22:11 Payload-Line-1
Payload-Line-2
Payload-Line-3

2015-06-05 16:22:12 Payload-Line-1

This is multiline start from same regex \d{1,4}\-\d{1,2}\-\d{1,2}, but i can not do this

Any python experts can help me?

via Chebli Mohamed

using pandas to read in a single column CSV

I am attempting to read in a single column CSV using the read_csv() function, but for some reason whenever I attempt to, i get an error:

ValueError: 'Message' is not in list

Here is the code that i am using (just a warning, I did not write all of this. I am inheriting this script from someone else and i have to maintain it.)

def get_data(fp, encoding='utf-8', columns=['Message']):
    """Get the columns from the raw data file provided."""
    if 'csv' in fp.split('.')[-1]:
        data = pd.read_csv(fp, usecols=columns,
                           converters={'Message':
                                       lambda x: unicode(x,
                                                     encoding,
                                                     errors='ignore')})

I am unsure why i am having this issue, and any help or pointing me in the right direction would be appreciated.

via Chebli Mohamed

mail.outbox django - Get the attached file for testing

I am sending a mail with an attached csv file. I am not storing the file on the system directly attaching it to the mail and sending it. I want to test the contents of the csv file.

Is there something like mail.outbox.file or something?

Thanks in advance!

via Chebli Mohamed

Project Euler 92

My code for solving problem 92 was correct but too slow and hence I tried to modify it by considering only one number for each possible permutation of that number, effectively reducing the size of the problem to 11439 from the original 10 million. Here's my code

import time
from Euler import multCoeff

start = time.time()

def newNum(n):
    return sum([int(dig)**2 for dig in str(n)])

def chain(n, l):
    if n in l:
        return n, l
    else:
        l.append(n)
        return chain(newNum(n), l)

nums = []

for i in range(1,10000000):
    if all(str(i)[j] <= str(i)[j+1] for j in range(len(str(i))-1)):
        nums.append(i)

count = 0   

for i in nums:
    if 89 in chain(i,[])[1]: 
        perms = multCoeff(i)
        count += perms

end = time.time() - start            

print count, end

multCoeff is a method that I created which is basically equivalent to len(set(permutations([int(j) for j in str(i)]))) and works just fine. Anyway, the problem is that the result I get is not the correct one, and it looks like I'm ignoring some of the cases but I can't really see which ones. I'd be really grateful if someone could point me in the right direction. Thanks.

via Chebli Mohamed

Get java version number from python

I need to get the java version number, for example "1.5", from python (or bash).

I would use:

os.system('java -version 2>&1 | grep "java version" | cut -d "\\\"" -f 2')

But that returns 1.5.0_30

It needs to be compatible if the number changes to "1.10" for example.

I would like to use cut or grep or even sed. It should be in one line.

via Chebli Mohamed

detection whether is an e-shop written in Magento

I'm trying to create a script which would do a simple thing. The input is a list of URLs and output is a list of those eshops which are written in Magento.

I've read that there is no way to realize whether is the eshop in Magento or something else but I've also read that there is a lot of signs that could tell you that this web page is using Magento almost 100% sure.

So I've found this page: magento detector which can tell you whether is it a Magento or not so I'm trying to use their information.

They say for example this:

Magento has its user interface files in a directory called /skin/. For the frontend (not the admin ui) the files are located in /skin/frontend. So if this directory exists in the page source then it is very likely that the store runs on Magento.

For example for this eshop: starkk told the detector that it is a magento and the one of condition it meets is the condition I've mentioned above.

How could I check whether the directory exists? I took a look on: http://ift.tt/1SHgVVO using browser but the page raises error.

And additional question: Do you know another and better way how to detect Magento?

via Chebli Mohamed

Print series of lists with aligned columns

I have a function that is passed a list of ints, and the desired row length. It is assumed that the length of the data is evenly divisible by the row length. All the ints in the list represent bytes, so they are no longer than 3 digits, and no smaller than 0. Here is my function:

    def pprint(data, rowlen):
    # Prints out the list in a nice grid format
    print '-' * 50
    for index in range(0, len(data), rowlen):
        print data[index:(index+rowlen)]
    print '-' * 50
    return data

The output looks something like this:

[0, 0, 2, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0, 0]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0, 0, 0, 0, 0, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0]

As you can see, this sort of works, but I'd like something more like this:

[0  , 0  , 2  , 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0  , 0  ]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0  , 0  , 0  , 0  , 0  , 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139]
[139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 139, 0  ]

That way, all the data is aligned in a grid format and it is easy to see where everything is.

Is there an easy way to achieve this without having to iterate through every entry and inserting spaces.

via Chebli Mohamed

TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given

I have hit a problem with one of the examples in the book Beginning Python Games Development Second Edition.

It has me setting up a vector class (full code at the end of the Q) with the following __init__ function:

def __init__(self, x=0, y=0):
    self.x = x
    self.y = y

It then asks me to run this code snippet against it:

from Vector2 import *

A = (10.0, 20,0)
B = (30.0, 35.0)
AB = Vector2.from_points(A, B)
step = AB * .1
position = Vector2(A, B)
step = AB * .1
print(*A)
position = Vector2(*A)
for n in range(10):
    position += step
    print(position)

The result is the following error:

Traceback (most recent call last):
  File "C:/Users/Charles Jr/Dropbox/Python/5-14 calculating positions.py", line 10, in <module>
    position = Vector2(*A)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given

When I put a print on the *A, it comes up as just 2 numbers, as you would expect. Why is it somehow turning this into 4?

Full Vector2 code:

import math

class Vector2:

    def __init__(self, x=0, y=0):
        self.x = x
        self.y = y

    def __str__(self):
        return "(%s, %s)"%(self.x, self.y)

    def from_points(P1, P2):
        print("foo")
        return Vector2( P2[0] - P1[0], P2[1] - P1[1])

    def get_magnitude(self):
        return math.sqrt( self.x**2 + self.y**2 )

    def normalise(self):
        magnitude = self.get_magnitude()
        self.x /= magnitude
        self.y /= magnitude

    # rhs stands for right hand side
    def __add__(self, rhs):
        return Vector2(self.x + rhs.x, self.y + rhs.y)

    def __sub__(self, rhs):
        return Vector2(self.x - rhs.x, self.y - rhs.y)

    def __neg__(self):
        return Vector2(-self.x, -self.y)

    def __mul__(self, scalar):
        return Vector2(self.x * scalar, self.y * scalar)

    def __truediv__(self, scalar):
        return Vector2(self.x / scalar, self.y / scalar)

via Chebli Mohamed

hash() in Python [duplicate]

This question already has an answer here:

• What does hash do in python? 4 answers

How does hash function works in Python? Why is hash('\0b') = hash('\0\0c') even though '\0b' is not equal to '\0\0c'?

>>>print hash('\0b') == hash('\0\0c')
True
>>>print hash('\0b'),hash('\0\0c'),hash('\0\0\0d'),hash('\0\0\0\0e')
96 96 96 96

via Chebli Mohamed

"Start External Program" Debugging option not appearing in VS

I'm trying to run my python script in debug mode via ArcMap. I tried to follow instructions here, but when I get to the debug window, I don't see and option for "Start External Program". What's up with that?

Btw, using VS'13.

via Chebli Mohamed

Execute Python script from Php

I have a PHP webpage on my raspberry pi with 2 buttons (on and off) The on button button redirects to On.php The off button redirects to Off.php In "/usr/lib/cgi-bin" I have a python script that I would like to execute (script.py) I can perfectly execute it from the terminal by typing

cd /usr/lib/cgi-bin
sudo python script.py

It works if I do it from the terminal.

The problem is the PHP file (On.php) in my "/var/www" folder. This is what I wrote:

<?php
exec('cd /usr/lib/cgi-bin');
exec('sudo python script.py');
?>

Why is the script executing from the terminal, but not from my PHP?

via Chebli Mohamed

python - unable to add image to the GUI(tkinter) on windows

I am using python( my version is 2.7 ). I want to add an image to GUI (Tkinter) and then convert into executable format using pyinstaller. I did followed as on SO, and also as said on ActiveState

When i mention the image's path on the code, it works only if i run it directly. If i convert it to exe it doesnt open.

Changing the code as mentioned from other solutions, like by converting it into encoded string, it runs fine on linux. But on windows it throws error

code:

from Tkinter import *
from PIL import ImageTk, Image

logo = '''
----- encoded string -----
'''

root = Tk()
logoimage = Tkinter.PhotoImage(master=root, data=logo)
Label(root, image=logoimage).pack()
root.mainloop()

Change 1: The above code works on linux. On windows i get error on the line logoimage = Tkinter.PhotoImage(master=root, data=logo) as

NameError: name 'Tkinter' is not defined

Change 2: So i tries changing the line as logoimage = ImageTk.PhotoImage(master=root, data=logo). The error i get is

File "C:\Python27\lib\site-packages\PIL\ImageTk.py", line 88, in __init__
    image = Image.open(BytesIO(kw["data"]))
  File "C:\Python27\lib\site-packages\PIL\Image.py", line 2330, in open
    % (filename if filename else fp))
IOError: cannot identify image file <_io.BytesIO object at 0x00000000024BB150>
Exception AttributeError: "'PhotoImage' object has no attribute '_PhotoImage__photo'" in <bound method PhotoImage.__del__ of <PIL.ImageTk.PhotoImage object at 0x00000000024D49E8>> ignored

Change 3: But, if i change the line as iconImage= ImageTk.PhotoImage(Image.open('path_to_image.png')). It works only if i run directly. If i convert it to executable, then console opens for 2-3 seconds and displaying error something like Unable to locate the image file

via Chebli Mohamed

What is a Pythonic way for Dependency Injection?

Introduction

For Java, Dependency Injection works as pure OOP, i.e. you provide an interface to be implemented and in your framework code accept an instance of a class that implements the defined interface.

Now for Python, you are able to do the same way, but I think that method was too much overhead right in case of Python. So then how would you implement it in the Pythonic way?

Use Case

Say this is the framework code:

class FrameworkClass():
    def __init__(self, ...):
        ...

    def do_the_job(self, ...):
        # some stuff
        # depending on some external function

The Basic Approach

The most naive (and maybe the best) way is to require the external function to be supplied into the FrameworkClass constructor, and then be invoked from the do_the_job method.

Framework Code:

class FrameworkClass():
    def __init__(self, func):
        self.func = func

    def do_the_job(self, ...):
        # some stuff
        self.func(...)

Client Code:

def my_func():
    # my implementation

framework_instance = FrameworkClass(my_func)
framework_instance.do_the_job(...)

Question

The question is short. Is there any better Pythonic way to do this?

via Chebli Mohamed

oh-my-zsh: What is the benefit of its python plugin?

The python plugin of on-my-zsh looks like:

  1 # Find python file
  2 alias pyfind='find . -name "*.py"'
  3
  4 # Remove python compiled byte-code in either current directory or in a
  5 # list of specified directories
  6 function pyclean() {
  7     ZSH_PYCLEAN_PLACES=${*:-'.'}
  8     find ${ZSH_PYCLEAN_PLACES} -type f -name "*.py[co]" -delete
  9     find ${ZSH_PYCLEAN_PLACES} -type d -name "__pycache__" -delete
 10 }
 11
 12 # Grep among .py files
 13 alias pygrep='grep --include="*.py"'

Does it mean that this plugin is only benefit for some alias ? Is there any amazing advantage?

via Chebli Mohamed

Pandas asign to slice

I have some old python code that I have to transition from 0.10 to 0.15 big jump. The old python code has multiple assignments to slices using the following structure:

condition = a['Column'] != 'option' #  with lots of complex logic
df.ColumnX[condition] = new_value

I know this is a an assigning to a slice. In pandas 10 this worked, as documented in various pandas upgrades it does not work. without having to refactor the code totally is the a way of changing line 2 so that it can still use the loc method while still using the condition as defined on line 1.

I have tried:

df.loc[:,condition] = 'new_val'

but I get

IndexingError: Unalignable boolean Series key provided

can anybody suggest a syntax that would (preferably backward compatible to pandas 0.10 that would provide quick fix without re-factoring dozens of complex selections?

via Chebli Mohamed

PyParsing - Parsing strings that must contain at least one occurrence of a symbol

I am trying to parse strings using PyParsing and detect at parsing time if they contain at least one occurrence of a defined symbol (say ‘$’).

For example, the following string would be parsed:

"('a$x' & 'b') | $ | ('c' <-> $)"

But this one would not (and a parsing exception would be raised):

"('a$x' & 'b') | 'c'"

How can this be done?

My goal here is to build abstract syntax trees (ASTs) that must contain at least one occurrence of this special symbol '$'. For the moment, I consider '$' as a simple literal:

my_symbol = Literal('$')
my_symbol.setParseAction(lambda tokens: Symbol())

and I verify the "at least one occurrence" condition after parsing, i.e., once my AST is built (note that Symbol() is a class that represent '$' in my AST). However, I think that a detection at parsing time would be more efficient.

EDIT: I don't know if it may help but here is a simplified version of the grammar I would like to parse:

<a> ::= '$' | '~' <a> | <a> '&' <b> | <b> '&' <a> | <a> '|' <b> 
      | <b> '|' <a> | <a> '->' <b> | <b> '->' <a> | <a> '<->' <b> 
      | <b> '<->' <a> | '(' <a> ')'

<b> ::= 'True' | 'False' | <atom> | '~' <b> | <b> '&' <b> | <b> '|' <b> 
      | <b> '->' <b> | <b> '<->' <b> | '(' <b> ')'

<atom> is defined by any string surrounded by single quotes

via Chebli Mohamed

How to generate a list of random numbers so their sum would be equal to a randomly chosen number

I want to generate a list of random distribution of numbers so their sum would be equal to a randomly chosen number. For example, if randomly chosen number is 5, the distribution would be [1 2 2] or [2 3] or [1 1 1 2] and so on. Any suggestions are welcome!

via Chebli Mohamed

Import module error with virtualenv

So I've recently installed virtualenv and tried using it but I can't get it to work. My env directory is on my Desktop and I've made a folder where I have my scripts but when I run them with IDLE I get an import module error. Same goes for when I run the script with the command line after activating the virtualenv of course. I also activate the virtualenv when trying to runt he scripts with IDLE. I've tried many things on stackoverflow but nothing seems to work for me.

Edit, yes I have the packages installed and I'm using Windows with Python 3.4.

via Chebli Mohamed

Create super task to inherit from for celery tasks

I am creating tasks using the @app.task decorator.

@app.task(name=settings.CELERY_TASK_MY_TASK)
def my_task(object_slug):

Is it possible to have those tasks inherit attributes and parameters from a "superclass" kind of task?

via Chebli Mohamed

Flask Permission Denied Opening File

As part of a Flask app I have, there's a page that opens a log file. When the app is run with the Flask debug server, the page works correctly. However, when run with Gunicorn, an exception is thrown:

IOError: [Errno 13] Permission denied: '/var/log/gunicorn/app.log'

I checked, and the path to the file is correct, and the permissions for the file are:

-rw-rw-rw- 1 root root 1.3M Aug 4 12:01 app.log

Shouldn't any application be able to read (and write to, for that matter) this file? I'm not sure what is causing this error.

via Chebli Mohamed

Install yum on SuSE 11.1 64 bit

I am trying to install pyodbc in order to connect to a database from Linux, however I found that there is no yum installed in the first place. On searching the internet, I found below are the RPMs required in order to install yum, however I am unable to find the right version. I checked this website, http://ift.tt/1IgAY3S, but I could not find the RPM python-sqlite. Could you give me some direction on this. Appreciate all the help.

via Chebli Mohamed

Django forms extra empty radio button

In rendering a model form an extra radio button is produced and I don't know where it's coming from:

>>> f = DocumentForm()
>>> print f['document_type']
<ul id="id_document_type">
<li><label for="id_document_type_0"><input checked="checked" id="id_document_type_0" name="document_type" type="radio" value="" /> ---------</label></li>
<li><label for="id_document_type_1"><input id="id_document_type_1" name="document_type" type="radio" value="1" /> Campus LAN</label></li>
<li><label for="id_document_type_2"><input id="id_document_type_2" name="document_type" type="radio" value="2" /> WAN</label></li>
<li><label for="id_document_type_3"><input id="id_document_type_3" name="document_type" type="radio" value="3" /> UC</label></li>
</ul>

That first radio button with value="" and the text as ---------, I've scoured my code and can't work out where it originates from?

models.py

class DocumentType(models.Model):
    name = models.CharField("Document Type", max_length=240)

class Document(models.Model):
    document_type = models.ForeignKey(DocumentType,
                                      verbose_name="Document Type")

>>> DocumentType.objects.all()
[<DocumentType: Campus LAN>, <DocumentType: WAN>, <DocumentType: UC>]
>>> d = Document.objects.all()
>>> for x in d:
...     print x.document_type
... 
Campus LAN
Campus LAN

via Chebli Mohamed

Wondering about APIViews in Django

Hi I'm working on a Django site and am trying to figure out what the previously implemented code does. I'm new to Django so just figuring this stuff out on the fly. I'm looking at resources.py, and there's a snippet of code that looks like this:

class BaseQueryView(APIView):

def _parse_filter_values(self, data):

   include = []

   exclude = []

   for d in data:

and so on

I guess what I'm wondering is where is the APIView getting the "data" from? How can I find out how "data" is structured?

via Chebli Mohamed

how to calculate the time interval in a list of lists using python?

I am a beginner of python and got stuck in a case to process data. I am thankful for any help.

The original list of lists is as following. The the second element in each list is a code standing for certain status of traffic signal light. The code '0','1','3' represent minimum green status of traffic light. I want to calculate the time duration of minimum green. The definition of one minimum_green duration is that the start time of first other status next to a minimum_green_status subtracts the start time of the minimum_green_status. Then, add the minimum green time in a list.

My stupid solution:

def get_minimum_green(sg_status):

    minimum_green_list = [] 
    start_time_green = None 
    for i in range(len(sg_status)):
        if not(sg_status[i][1] == '3' or sg_status[i][1] == '1' or sg_status[i][1] == '0' and (i != 0 or sg_status[i-1][1] != '3' and sg_status[i-1][1] != '1' and sg_status != '0')) :
            continue
        else:
            start_time_green = sg_status[i][0]
            print "start: "+str(start_time_green)
            for j in range(i,len(sg_status)):
                if not(sg_status[j][1] != '3' and sg_status[j][1] != '1' and sg_status[j][1] != '0' and (sg_status[j-1][1] == '3' or sg_status[j-1][1] == '1' or sg_status == '0')):
                    continue
                else:
                    end_time_green = sg_status[j][0]
                    print "end  :" + str(end_time_green)
                    minumum_green = end_time_green - start_time_green
                    minimum_green_list.append(minumum_green) 
                    break 

    return str(minimum_green_list) 

But the result is obviously wrong and the amount of start_time_green and end_time_green are even different.

sg_status =[[[datetime.datetime(2015, 7, 8, 3, 0, 24, 302000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 0, 24, 903000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 0, 27, 600000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 0, 28, 201000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 0, 37, 500000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 0, 38, 101000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 40, 500000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 41, 1000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 50, 600000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 50, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 51, 502000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 55, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 55, 901000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 57, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 1, 58, 703000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '<'], [datetime.datetime(2015, 7, 8, 3, 2, 1, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'A'], [datetime.datetime(2015, 7, 8, 3, 2, 2, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'A'], [datetime.datetime(2015, 7, 8, 3, 2, 3, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 7, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 10, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 11, 802000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 12, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 12, 801000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 13, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 14, 703000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 15, 704000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 16, 705000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 16, 806000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 18, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 20, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 21, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 22, 903000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 23, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 24, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 25, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 26, 703000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 2, 26, 804000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '0'], [datetime.datetime(2015, 7, 8, 3, 2, 27, 805000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '1'], [datetime.datetime(2015, 7, 8, 3, 2, 30, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '1'], [datetime.datetime(2015, 7, 8, 3, 2, 32, 801000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 2, 35, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 3, 51, 400000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 3, 52, 1000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 3, 58, 500000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 3, 59, 101000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 17, 900000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 18, 401000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 24, 800000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 25, 301000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 25, 902000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 26, 603000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 28, 704000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 29, 305000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 35, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 5, 35, 601000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 12, 100000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 13, 1000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 15, 402000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 15, 903000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 17, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 19, 1000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 19, 402000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 23, 800000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 24, 501000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 25, 802000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 26, 303000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 26, 704000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 27, 205000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 27, 906000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 28, 500000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 30, 801000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 31, 302000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 33, 200000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 33, 901000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 37, 402000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 38, 100000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 49, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 54, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 56, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '3'], [datetime.datetime(2015, 7, 8, 3, 8, 57, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), '<'], [datetime.datetime(2015, 7, 8, 3, 9, 0, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'A'], [datetime.datetime(2015, 7, 8, 3, 9, 1, 701000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'A'], [datetime.datetime(2015, 7, 8, 3, 9, 2, 702000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 9, 6, 700000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 9, 7, 301000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 9, 8, 402000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 9, 9, 203000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'], [datetime.datetime(2015, 7, 8, 3, 9, 9, 704000, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=180, name=None)), 'B'],]

via Chebli Mohamed

Boolean field query with sqlalchemy

Postgres model:

class Song(db.Model):
    id3_parsed = db.Column(db.Boolean, server_default=u'false')

Running the following query gives the correct count:

select count(*) from song where id3_parsed is false;

But how do I do it with flask-sqlalchemy? This doesn't work:

songs = Song.query.filter(Song.id3_parsed == False).all()

via Chebli Mohamed

Django Logical Deletes on Models

So i decided to fork pinax-models because i need the capability of soft deleting objects and their related (setting a flag but not really deleting them). This is a project that seems to be unmaintained , so i diged into the code and i am trying to make some improvements.

It basically fetched the objects, looks for its related and then saves the date_removed to fake the deletion.

It still has some issues and one of them which i am trying to solve is that it always "cascades" the deletes of the related but it should only do it if the related field is not nullable. if its nullable then it should not delete the related field.

I have 2 question regarding this matter:

1- How can i improve this code below to support that feature? I know it has something to do with _meta, thought about using obj._meta.get_fields_with_model but i am afraid it does not do options.concrete_fields + options.many_to_many + options.virtual_fields. i was thinking about getting all the existing fields in a model an then evaluate if its a foreign key to a certain model class and then check if obj._meta.get_field('something').null. Is this the right approach??

2 - The original author is using self._meta.get_all_related_objects() but in the _meta documentation i saw other functions like for example get_all_related_many_to_many_objects so do i need to add that validation to the code as well or self._meta.get_all_related_objects() is enough ??

I am trying to improve this and push it to my pinax-models fork so any help would be appreciated.

code:

class CustomLogicalDeleteModel(LogicalDeleteModel):

    def delete(self):
        # Fetch related models
        related_objs = [
            relation.get_accessor_name()
            for relation in self._meta.get_all_related_objects()
        ]

        for objs_model in related_objs:

            if hasattr(self, objs_model):
                local_objs_model = getattr(self, objs_model)
                if hasattr(local_objs_model, 'all'):
                    # Retrieve all related objects
                    objs = local_objs_model.all()

                    for obj in objs:
                        # Checking if inherits from logicaldelete
                        if not issubclass(obj.__class__, LogicalDeleteModel):
                            break
                        obj.delete()
                else:
                    obj = local_objs_model

                    if not issubclass(obj.__class__, LogicalDeleteModel):
                        break
                    obj.delete()

        # Soft delete the object
        self.date_removed = datetime.datetime.now()
        self.save()

    class Meta:
        abstract = True

via Chebli Mohamed

How can i access the vmware machine for system logs using Python script

Folks,

i have a Linux machine running on VmWare configured on my Mac...

how can i read Linux system logs using python from my local machine?

via Chebli Mohamed

How to free resource in PyGame mixer?

I use gTTS python module to get mp3 from Google Text-To-Speech API and PyGame to play output mp3 files without opening external player (is there any simpler way to do it?)

However it seems like PyGame mixer doesn't free file resource, even after it's quit method.

phrase = "Hello!"
tts = gtts.gTTS(text=phrase, lang='en')
tts.save("googleTTS.mp3")

f = MP3("googleTTS.mp3")
mixer.init(f.info.sample_rate)
mixer.music.load("googleTTS.mp3")
mixer.music.play()
while mixer.music.get_busy() == True:
    continue
mixer.quit()        # doesn't free resource?

phrase = "Bye!"
tts = gtts.gTTS(text=phrase, lang='en')
tts.save("googleTTS.mp3")

Last line gives exception:

    IOError: [Errno 13] Permission denied: 'googleTTS.mp3'

I should notice that the problem isn't in tts.save function, cause code without mixer works fine.

How can I free mixer resource and use the same file over and over again?

via Chebli Mohamed

How to correctly add Foreign Key constraints to SQLite DB using SQLAlchemy

I'm very new to SQLAlchemy and I'm trying to figure it out.

Please have in mind the following test setup:

class Nine(Base):
    __tablename__ = 'nine'
    __table_args__ = (sqlalchemy.sql.schema.UniqueConstraint('nine_b', name='uq_nine_b'), )

    nine_a = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), primary_key=True, autoincrement=False, nullable=False)
    nine_b = sqlalchemy.Column(sqlalchemy.String(20), nullable=False)


class Seven(Base):
    __tablename__ = 'seven'
    __table_args__ = (sqlalchemy.sql.schema.PrimaryKeyConstraint('seven_a', 'seven_b'),
                      sqlalchemy.sql.schema.Index('fk_seven_c_nine_a_idx', 'seven_c'),)

    seven_a = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), nullable=False)
    seven_b = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), nullable=False)
    seven_c = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), sqlalchemy.ForeignKey('nine.nine_a'), nullable=False)
    seven_d = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), nullable=False)

    nine = sqlalchemy.orm.relationship(Nine, backref=sqlalchemy.orm.backref('seven'), uselist=False)


class Three(Base):
    __tablename__ = 'three'
    __table_args__ = (sqlalchemy.sql.schema.UniqueConstraint('three_b', 'three_c', name='uq_three_b_c'),
                      sqlalchemy.sql.schema.Index('fk_three_c_seven_a_idx', 'three_c'), )

    three_a = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), primary_key=True, autoincrement=True, nullable=False)
    three_b = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), nullable=False)
    three_c = sqlalchemy.Column(sqlalchemy.dialects.sqlite.INTEGER(), sqlalchemy.ForeignKey('seven.seven_a'), nullable=False)

    seven = sqlalchemy.orm.relationship(Seven, backref=sqlalchemy.orm.backref('three'), uselist=False)

That translates into the following DDLs:

CREATE TABLE nine (
    nine_a INTEGER NOT NULL, 
    nine_b VARCHAR(20) NOT NULL, 
    PRIMARY KEY (nine_a), 
    CONSTRAINT uq_nine_b UNIQUE (nine_b)
);

CREATE TABLE seven (
    seven_a INTEGER NOT NULL, 
    seven_b INTEGER NOT NULL, 
    seven_c INTEGER NOT NULL, 
    seven_d INTEGER NOT NULL, 
    PRIMARY KEY (seven_a, seven_b), 
    FOREIGN KEY(seven_c) REFERENCES nine (nine_a)
);

CREATE INDEX fk_seven_c_nine_a_idx ON seven (seven_c);

CREATE TABLE three (
    three_a INTEGER NOT NULL, 
    three_b INTEGER NOT NULL, 
    three_c INTEGER NOT NULL, 
    PRIMARY KEY (three_a), 
    CONSTRAINT uq_three_b_c UNIQUE (three_b, three_c), 
    FOREIGN KEY(three_c) REFERENCES seven (seven_a)
);

CREATE INDEX fk_three_c_seven_a_idx ON three (three_c);

All tables are empty. Then, the following code statements:

session.add(Nine(nine_a=1, nine_b='something'))
session.add(Nine(nine_a=2, nine_b='something else'))
session.commit()

session.add(Seven(seven_a=7, seven_b=7, seven_c=7, seven_d=7))
session.commit()

session.add(Three(three_a=3, three_b=3, three_c=3))
sessionDB.commit()

Can somebody please explain why is the above code snippet executing without errors? Should't the FK constraints stop from inserting a new row into seven or three? I assume there is something wrong with how the FKs are described in the classes themselves, but I don't know where the problem is (and how to fix it).

[Edit 1]

Adding __table_args__ for all classes (forgot to include them).

[Edit 2]

Adding DDLs for further reference.

via Chebli Mohamed

CherryPy with Bottle sometimes doesn't reload

I'm using the Bottle framework, CherryPy, and nginx doing reverse proxy. I have a problem where randomly autoreload won't restart the process but does kill off the old one, thus leaving the site down. I'm new to hosting on Linux, coding in Python, using CherryPy, and using nginx.

Is there a default log some place (I'm running Ubuntu server) that might have information as to why it doesn't restart? Or something I can enable to figure out why? It's been awesome otherwise, just this one problem and it seems to happen completely randomly.

Here's what my CherryPy code looks like:

import cherrypy as cp
from bottle import Bottle

APP = Bottle()

def run(app, host='0.0.0.0', port=8090, **config):
    cp.tree.graft(app, '/')
    cp.config.update(config)
    cp.config.update({
        'server.socket_port': port,
        'server.socket_host': host,
    })
    cp.engine.signals.subscribe()
    cp.engine.start()
    cp.engine.block()


if __name__ == '__main__':
    run(APP)

I'm not sure if this is the optimal way. I found this on a different stackoverflow question.

via Chebli Mohamed

Python Concat Two Dictionaries with comma

If I have two dicts in python

d1={1:2,3:4}
d2={5:6,7:9}

How can I combine that to make

d2 = {{1:2,3:4}, {5:6,7:9}}

via Chebli Mohamed

python: globally reloading a module

I am making a pygame dialog and i have files that are like that:

window.py:

class Window():
    def set_size(self, size):
        self.size = size

main_window = Window()

dialogs.py:

from window import main_window
class Question():
    def __init__(self):
        # lots of initializing
        self.position = (main_window.size[0] / 2, main_window.size[1] / 2) # centering dialog

question = Question()

other files:

from window import main_window
from dialogs import question

# do something

The problem is that if i change the size of the main window in one of the other files, then the question dialog will no longer be centered, since question is already initialized with a different screen size. So, i want to make the set_size method reload the question module for all other files. How can i do this?

Thanks

via Chebli Mohamed