Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 878 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0274. Min Act: -0.0004

Data Index: 665729 (C4 (Web Text))

Max Activating Token Index: 954

Text #1

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0278. Min Act: -0.0004

Data Index: 1334761 (C4 (Web Text))

Max Activating Token Index: 891

Text #2

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0270. Min Act: -0.0004

Data Index: 1353107 (C4 (Web Text))

Max Activating Token Index: 128

Text #3

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0266. Min Act: -0.0004

Data Index: 32914 (C4 (Web Text))

Max Activating Token Index: 663

Text #4

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0253. Min Act: -0.0004

Data Index: 260676 (C4 (Web Text))

Max Activating Token Index: 760

Text #5

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0252. Min Act: -0.0004

Data Index: 947721 (C4 (Web Text))

Max Activating Token Index: 512

Text #6

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0256. Min Act: -0.0004

Data Index: 890199 (C4 (Web Text))

Max Activating Token Index: 49

Text #7

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0259. Min Act: -0.0004

Data Index: 923812 (C4 (Web Text))

Max Activating Token Index: 911

Text #8

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0248. Min Act: -0.0004

Data Index: 511397 (C4 (Web Text))

Max Activating Token Index: 533

Text #9

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0252. Min Act: -0.0004

Data Index: 1164799 (C4 (Web Text))

Max Activating Token Index: 379

Text #10

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0251. Min Act: -0.0004

Data Index: 169298 (C4 (Web Text))

Max Activating Token Index: 8

Text #11

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0244. Min Act: -0.0004

Data Index: 946003 (C4 (Web Text))

Max Activating Token Index: 142

Text #12

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0247. Min Act: -0.0004

Data Index: 178610 (C4 (Web Text))

Max Activating Token Index: 4

Text #13

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0246. Min Act: -0.0004

Data Index: 623110 (C4 (Web Text))

Max Activating Token Index: 120

Text #14

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0246. Min Act: -0.0004

Data Index: 1117919 (C4 (Web Text))

Max Activating Token Index: 860

Text #15

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0245. Min Act: -0.0004

Data Index: 131968 (C4 (Web Text))

Max Activating Token Index: 760

Text #16

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0240. Min Act: -0.0004

Data Index: 914398 (C4 (Web Text))

Max Activating Token Index: 804

Text #17

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0244. Min Act: -0.0004

Data Index: 899537 (C4 (Web Text))

Max Activating Token Index: 181

Text #18

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0244. Min Act: -0.0004

Data Index: 233630 (C4 (Web Text))

Max Activating Token Index: 73

Text #19

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0240. Min Act: -0.0004

Data Index: 1025218 (C4 (Web Text))

Max Activating Token Index: 850