python - Appending rows to csv using dict_writer -

i appending rows existing csv file following code:

counter=1 thepage in newpages:         web_page = urllib2.urlopen(thepage)     soup = beautifulsoup(web_page.read())      fieldnames=["var1", "var2","var3","var4","var5", "var6", "var7"]     if counter==1:             f = open('file.csv', 'wb')               my_writer = csv.dictwriter(f, fieldnames)             my_writer.writeheader()             f.close()      variables={ele:"missing" ele in fieldnames}     variables['var1']=str(counter)     variables['var2']=soup.find_all('strong')[0].text     variables['var3']=soup.find_all('p')[1].text[0]     variables['var4']=soup.find_all('p')[1].text[1]     variables['var4']=soup.find_all('p')[2].text     variables['var6']=soup.find_all('p')[6].text     variables['var7']=soup.find_all('p')[7].text     print variables     open('file.csv', 'r+b') f:         header = next(csv.reader(f))         dict_writer = csv.dictwriter(f, header)         dict_writer.writerow(variables)     counter+=1 

where "variables" dictionary created within loop.the loops continue running no problem when @ csv file had stopped appending rows more or less in 30th row. checked , not problem "variables" dictionary , when change file , use other variables fails more or less in same row. problem?

the problem is, think, overwriting file each time. being opened mode 'r+b' , positions file pointer beginning of file. subsequent writes therefore overwrite previous contents of file.

you can open file in append mode can done using mode 'ab'.

having said that, not necessary continually open file, nor output csv headers within body of loop. suggest following:

import csv  counter=1 fieldnames=["var1", "var2","var3","var4","var5", "var6", "var7"]  open('file.csv', 'wb') f:         my_writer = csv.dictwriter(f, fieldnames)     my_writer.writeheader()      thepage in newpages:             web_page = urllib2.urlopen(thepage)         soup = beautifulsoup(web_page.read())          variables={ele:"missing" ele in fieldnames}         variables['var1']=str(counter)         variables['var2']=soup.find_all('strong')[0].text         variables['var3']=soup.find_all('p')[1].text[0]         variables['var4']=soup.find_all('p')[1].text[1]         variables['var4']=soup.find_all('p')[2].text         variables['var6']=soup.find_all('p')[6].text         variables['var7']=soup.find_all('p')[7].text         print variables         my_writer.writerow(variables)          counter+=1