Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4399 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0069. Min Range: -3.0069

Max Act: 3.0069. Min Act: -0.0000

Data Index: 1073559 (C4 (Web Text))

Max Activating Token Index: 520

Text #1

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.8397. Min Act: -0.0000

Data Index: 1040790 (C4 (Web Text))

Max Activating Token Index: 988

Text #2

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.8091. Min Act: -0.0000

Data Index: 741317 (C4 (Web Text))

Max Activating Token Index: 988

Text #3

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.5847. Min Act: -0.0000

Data Index: 1076882 (C4 (Web Text))

Max Activating Token Index: 142

Text #4

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.2092. Min Act: -0.0000

Data Index: 1179737 (C4 (Web Text))

Max Activating Token Index: 678

Text #5

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.1890. Min Act: -0.0000

Data Index: 836668 (C4 (Web Text))

Max Activating Token Index: 789

Text #6

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.0401. Min Act: -0.0000

Data Index: 1007942 (C4 (Web Text))

Max Activating Token Index: 56

Text #7

Max Range: 3.0069. Min Range: -3.0069

Max Act: 2.1183. Min Act: -0.0000

Data Index: 948871 (C4 (Web Text))

Max Activating Token Index: 696

Text #8

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.6927. Min Act: -0.0000

Data Index: 465987 (C4 (Web Text))

Max Activating Token Index: 512

Text #9

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.6976. Min Act: -0.0000

Data Index: 416560 (C4 (Web Text))

Max Activating Token Index: 604

Text #10

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.6652. Min Act: -0.0000

Data Index: 898681 (C4 (Web Text))

Max Activating Token Index: 934

Text #11

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.6133. Min Act: -0.0000

Data Index: 1087955 (C4 (Web Text))

Max Activating Token Index: 976

Text #12

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.5823. Min Act: -0.0000

Data Index: 447131 (C4 (Web Text))

Max Activating Token Index: 139

Text #13

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.6537. Min Act: -0.0000

Data Index: 21678 (C4 (Web Text))

Max Activating Token Index: 850

Text #14

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.4336. Min Act: -0.0000

Data Index: 289443 (C4 (Web Text))

Max Activating Token Index: 1013

Text #15

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.4194. Min Act: 0.0000

Data Index: 1262172 (C4 (Web Text))

Max Activating Token Index: 923

Text #16

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.4873. Min Act: -0.0000

Data Index: 174815 (C4 (Web Text))

Max Activating Token Index: 515

Text #17

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.4270. Min Act: -0.0000

Data Index: 612336 (C4 (Web Text))

Max Activating Token Index: 890

Text #18

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.3915. Min Act: -0.0000

Data Index: 360386 (C4 (Web Text))

Max Activating Token Index: 88

Text #19

Max Range: 3.0069. Min Range: -3.0069

Max Act: 1.3708. Min Act: -0.0000

Data Index: 1269511 (C4 (Web Text))

Max Activating Token Index: 683