Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1689 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.6819. Min Act: -0.0002

Data Index: 1582814 (Python Code)

Max Activating Token Index: 661

Text #1

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.6039. Min Act: -0.0002

Data Index: 1621238 (Python Code)

Max Activating Token Index: 970

Text #2

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.5332. Min Act: -0.0002

Data Index: 1539113 (Python Code)

Max Activating Token Index: 494

Text #3

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4910. Min Act: -0.0002

Data Index: 1382825 (Python Code)

Max Activating Token Index: 665

Text #4

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4936. Min Act: -0.0002

Data Index: 1528721 (Python Code)

Max Activating Token Index: 334

Text #5

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.5225. Min Act: -0.0002

Data Index: 1367819 (Python Code)

Max Activating Token Index: 496

Text #6

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4902. Min Act: -0.0002

Data Index: 1493258 (Python Code)

Max Activating Token Index: 591

Text #7

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.5081. Min Act: -0.0002

Data Index: 1578027 (Python Code)

Max Activating Token Index: 571

Text #8

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4989. Min Act: -0.0002

Data Index: 1579623 (Python Code)

Max Activating Token Index: 918

Text #9

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.5130. Min Act: -0.0002

Data Index: 1415060 (Python Code)

Max Activating Token Index: 339

Text #10

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4534. Min Act: -0.0002

Data Index: 1398039 (Python Code)

Max Activating Token Index: 819

Text #11

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4771. Min Act: -0.0002

Data Index: 1439421 (Python Code)

Max Activating Token Index: 453

Text #12

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4956. Min Act: -0.0002

Data Index: 1594079 (Python Code)

Max Activating Token Index: 649

Text #13

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.5238. Min Act: -0.0002

Data Index: 1549843 (Python Code)

Max Activating Token Index: 706

Text #14

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4896. Min Act: -0.0001

Data Index: 1621882 (Python Code)

Max Activating Token Index: 198

Text #15

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4877. Min Act: -0.0002

Data Index: 1644684 (Python Code)

Max Activating Token Index: 949

Text #16

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4541. Min Act: -0.0002

Data Index: 1634938 (Python Code)

Max Activating Token Index: 575

Text #17

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4720. Min Act: -0.0002

Data Index: 1522463 (Python Code)

Max Activating Token Index: 263

Text #18

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4670. Min Act: -0.0002

Data Index: 1463090 (Python Code)

Max Activating Token Index: 416

Text #19

Max Range: 0.6819. Min Range: -0.6819

Max Act: 0.4491. Min Act: -0.0002

Data Index: 1567606 (Python Code)

Max Activating Token Index: 586