Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1368 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4236. Min Range: -3.4236

Max Act: 3.4236. Min Act: -0.0000

Data Index: 461735 (C4 (Web Text))

Max Activating Token Index: 912

Text #1

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.9187. Min Act: -0.0000

Data Index: 469369 (C4 (Web Text))

Max Activating Token Index: 148

Text #2

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.8717. Min Act: -0.0000

Data Index: 869964 (C4 (Web Text))

Max Activating Token Index: 360

Text #3

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.7510. Min Act: -0.0000

Data Index: 1091283 (C4 (Web Text))

Max Activating Token Index: 394

Text #4

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.7052. Min Act: -0.0000

Data Index: 570688 (C4 (Web Text))

Max Activating Token Index: 591

Text #5

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.5756. Min Act: -0.0000

Data Index: 1095234 (C4 (Web Text))

Max Activating Token Index: 369

Text #6

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.5676. Min Act: -0.0000

Data Index: 761213 (C4 (Web Text))

Max Activating Token Index: 607

Text #7

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.6861. Min Act: -0.0000

Data Index: 962786 (C4 (Web Text))

Max Activating Token Index: 405

Text #8

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.6648. Min Act: -0.0000

Data Index: 38780 (C4 (Web Text))

Max Activating Token Index: 165

Text #9

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.5405. Min Act: -0.0000

Data Index: 1286523 (C4 (Web Text))

Max Activating Token Index: 874

Text #10

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.4980. Min Act: -0.0000

Data Index: 69206 (C4 (Web Text))

Max Activating Token Index: 211

Text #11

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.3673. Min Act: -0.0000

Data Index: 1056113 (C4 (Web Text))

Max Activating Token Index: 368

Text #12

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.4382. Min Act: -0.0000

Data Index: 399630 (C4 (Web Text))

Max Activating Token Index: 408

Text #13

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.4091. Min Act: -0.0000

Data Index: 664418 (C4 (Web Text))

Max Activating Token Index: 340

Text #14

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.5092. Min Act: -0.0000

Data Index: 168435 (C4 (Web Text))

Max Activating Token Index: 778

Text #15

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.4832. Min Act: -0.0000

Data Index: 345196 (C4 (Web Text))

Max Activating Token Index: 479

Text #16

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.3661. Min Act: -0.0000

Data Index: 1080485 (C4 (Web Text))

Max Activating Token Index: 665

Text #17

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.3249. Min Act: -0.0000

Data Index: 206925 (C4 (Web Text))

Max Activating Token Index: 869

Text #18

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.5508. Min Act: -0.0000

Data Index: 273451 (C4 (Web Text))

Max Activating Token Index: 312

Text #19

Max Range: 3.4236. Min Range: -3.4236

Max Act: 2.4506. Min Act: -0.0000

Data Index: 385099 (C4 (Web Text))

Max Activating Token Index: 817