Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4896 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.6134. Min Act: -0.0001

Data Index: 1224432 (C4 (Web Text))

Max Activating Token Index: 391

Text #1

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.5454. Min Act: -0.0001

Data Index: 1530401 (Python Code)

Max Activating Token Index: 6

Text #2

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.5255. Min Act: -0.0001

Data Index: 1282685 (C4 (Web Text))

Max Activating Token Index: 11

Text #3

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.5294. Min Act: -0.0001

Data Index: 646211 (C4 (Web Text))

Max Activating Token Index: 882

Text #4

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4755. Min Act: -0.0001

Data Index: 16387 (C4 (Web Text))

Max Activating Token Index: 157

Text #5

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.5191. Min Act: -0.0001

Data Index: 1013951 (C4 (Web Text))

Max Activating Token Index: 1009

Text #6

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4873. Min Act: -0.0001

Data Index: 532610 (C4 (Web Text))

Max Activating Token Index: 831

Text #7

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4816. Min Act: -0.0001

Data Index: 389935 (C4 (Web Text))

Max Activating Token Index: 836

Text #8

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4637. Min Act: -0.0001

Data Index: 617912 (C4 (Web Text))

Max Activating Token Index: 113

Text #9

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4952. Min Act: -0.0001

Data Index: 1092270 (C4 (Web Text))

Max Activating Token Index: 508

Text #10

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4945. Min Act: -0.0001

Data Index: 114427 (C4 (Web Text))

Max Activating Token Index: 985

Text #11

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.5037. Min Act: -0.0001

Data Index: 984521 (C4 (Web Text))

Max Activating Token Index: 239

Text #12

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4658. Min Act: -0.0001

Data Index: 1467654 (Python Code)

Max Activating Token Index: 900

Text #13

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4454. Min Act: -0.0001

Data Index: 1467880 (Python Code)

Max Activating Token Index: 3

Text #14

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4709. Min Act: -0.0001

Data Index: 1331244 (C4 (Web Text))

Max Activating Token Index: 918

Text #15

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4564. Min Act: -0.0001

Data Index: 81434 (C4 (Web Text))

Max Activating Token Index: 745

Text #16

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4852. Min Act: -0.0001

Data Index: 646962 (C4 (Web Text))

Max Activating Token Index: 868

Text #17

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4376. Min Act: -0.0001

Data Index: 1110330 (C4 (Web Text))

Max Activating Token Index: 841

Text #18

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4560. Min Act: -0.0001

Data Index: 1237304 (C4 (Web Text))

Max Activating Token Index: 667

Text #19

Max Range: 0.6134. Min Range: -0.6134

Max Act: 0.4502. Min Act: -0.0001

Data Index: 1531637 (Python Code)

Max Activating Token Index: 5