Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1128 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5401. Min Range: -3.5401

Max Act: 3.5401. Min Act: 0.0000

Data Index: 402599 (C4 (Web Text))

Max Activating Token Index: 789

Text #1

Max Range: 3.5401. Min Range: -3.5401

Max Act: 3.2278. Min Act: -0.0000

Data Index: 1064604 (C4 (Web Text))

Max Activating Token Index: 824

Text #2

Max Range: 3.5401. Min Range: -3.5401

Max Act: 2.9967. Min Act: -0.0000

Data Index: 222196 (C4 (Web Text))

Max Activating Token Index: 899

Text #3

Max Range: 3.5401. Min Range: -3.5401

Max Act: 2.9444. Min Act: 0.0000

Data Index: 1145705 (C4 (Web Text))

Max Activating Token Index: 928

Text #4

Max Range: 3.5401. Min Range: -3.5401

Max Act: 2.1561. Min Act: -0.0000

Data Index: 1348801 (C4 (Web Text))

Max Activating Token Index: 130

Text #5

Max Range: 3.5401. Min Range: -3.5401

Max Act: 2.0320. Min Act: -0.0000

Data Index: 762281 (C4 (Web Text))

Max Activating Token Index: 920

Text #6

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.9190. Min Act: -0.0000

Data Index: 1127794 (C4 (Web Text))

Max Activating Token Index: 322

Text #7

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.9090. Min Act: -0.0000

Data Index: 917951 (C4 (Web Text))

Max Activating Token Index: 461

Text #8

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.8649. Min Act: -0.0000

Data Index: 521869 (C4 (Web Text))

Max Activating Token Index: 371

Text #9

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.8496. Min Act: -0.0000

Data Index: 1324577 (C4 (Web Text))

Max Activating Token Index: 262

Text #10

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.8484. Min Act: -0.0000

Data Index: 62045 (C4 (Web Text))

Max Activating Token Index: 380

Text #11

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.6492. Min Act: -0.0000

Data Index: 1334312 (C4 (Web Text))

Max Activating Token Index: 740

Text #12

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.7021. Min Act: -0.0000

Data Index: 1316277 (C4 (Web Text))

Max Activating Token Index: 879

Text #13

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.8020. Min Act: -0.0000

Data Index: 1324991 (C4 (Web Text))

Max Activating Token Index: 315

Text #14

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.6960. Min Act: -0.0000

Data Index: 442924 (C4 (Web Text))

Max Activating Token Index: 655

Text #15

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.6738. Min Act: -0.0000

Data Index: 896640 (C4 (Web Text))

Max Activating Token Index: 655

Text #16

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.6495. Min Act: -0.0000

Data Index: 751635 (C4 (Web Text))

Max Activating Token Index: 897

Text #17

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.5489. Min Act: -0.0000

Data Index: 1069000 (C4 (Web Text))

Max Activating Token Index: 332

Text #18

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.5134. Min Act: -0.0000

Data Index: 863513 (C4 (Web Text))

Max Activating Token Index: 918

Text #19

Max Range: 3.5401. Min Range: -3.5401

Max Act: 1.5031. Min Act: -0.0000

Data Index: 203557 (C4 (Web Text))

Max Activating Token Index: 66