Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1297 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0377. Min Act: -0.0004

Data Index: 1103785 (C4 (Web Text))

Max Activating Token Index: 222

Text #1

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0370. Min Act: -0.0004

Data Index: 289310 (C4 (Web Text))

Max Activating Token Index: 64

Text #2

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0359. Min Act: -0.0004

Data Index: 204240 (C4 (Web Text))

Max Activating Token Index: 935

Text #3

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0357. Min Act: -0.0004

Data Index: 395760 (C4 (Web Text))

Max Activating Token Index: 603

Text #4

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0346. Min Act: -0.0004

Data Index: 1301231 (C4 (Web Text))

Max Activating Token Index: 645

Text #5

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0334. Min Act: -0.0004

Data Index: 1318485 (C4 (Web Text))

Max Activating Token Index: 766

Text #6

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0333. Min Act: -0.0004

Data Index: 187471 (C4 (Web Text))

Max Activating Token Index: 489

Text #7

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0336. Min Act: -0.0004

Data Index: 536357 (C4 (Web Text))

Max Activating Token Index: 439

Text #8

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0334. Min Act: -0.0004

Data Index: 243614 (C4 (Web Text))

Max Activating Token Index: 639

Text #9

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0331. Min Act: -0.0004

Data Index: 1116482 (C4 (Web Text))

Max Activating Token Index: 255

Text #10

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0322. Min Act: -0.0004

Data Index: 524101 (C4 (Web Text))

Max Activating Token Index: 659

Text #11

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0328. Min Act: -0.0004

Data Index: 1268116 (C4 (Web Text))

Max Activating Token Index: 103

Text #12

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0311. Min Act: -0.0004

Data Index: 1085245 (C4 (Web Text))

Max Activating Token Index: 872

Text #13

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0317. Min Act: -0.0004

Data Index: 1107732 (C4 (Web Text))

Max Activating Token Index: 536

Text #14

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0316. Min Act: -0.0004

Data Index: 153863 (C4 (Web Text))

Max Activating Token Index: 802

Text #15

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0315. Min Act: -0.0004

Data Index: 557842 (C4 (Web Text))

Max Activating Token Index: 100

Text #16

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0306. Min Act: -0.0004

Data Index: 619383 (C4 (Web Text))

Max Activating Token Index: 658

Text #17

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0311. Min Act: -0.0004

Data Index: 15529 (C4 (Web Text))

Max Activating Token Index: 819

Text #18

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0309. Min Act: -0.0004

Data Index: 991221 (C4 (Web Text))

Max Activating Token Index: 537

Text #19

Max Range: 0.0377. Min Range: -0.0377

Max Act: 0.0307. Min Act: -0.0004

Data Index: 1059203 (C4 (Web Text))

Max Activating Token Index: 775