Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1093 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.1598. Min Act: -0.0000

Data Index: 267151 (C4 (Web Text))

Max Activating Token Index: 724

Text #1

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0890. Min Act: -0.0000

Data Index: 1291116 (C4 (Web Text))

Max Activating Token Index: 513

Text #2

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0885. Min Act: -0.0000

Data Index: 1313220 (C4 (Web Text))

Max Activating Token Index: 876

Text #3

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.6633. Min Act: -0.0000

Data Index: 844767 (C4 (Web Text))

Max Activating Token Index: 305

Text #4

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.6695. Min Act: -0.0000

Data Index: 662888 (C4 (Web Text))

Max Activating Token Index: 319

Text #5

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.5485. Min Act: -0.0000

Data Index: 121879 (C4 (Web Text))

Max Activating Token Index: 425

Text #6

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.5354. Min Act: -0.0000

Data Index: 1274765 (C4 (Web Text))

Max Activating Token Index: 367

Text #7

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3991. Min Act: -0.0000

Data Index: 1114592 (C4 (Web Text))

Max Activating Token Index: 12

Text #8

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.4079. Min Act: -0.0000

Data Index: 1282611 (C4 (Web Text))

Max Activating Token Index: 160

Text #9

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3679. Min Act: -0.0000

Data Index: 1224895 (C4 (Web Text))

Max Activating Token Index: 897

Text #10

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3744. Min Act: -0.0000

Data Index: 424785 (C4 (Web Text))

Max Activating Token Index: 521

Text #11

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3450. Min Act: -0.0000

Data Index: 1202718 (C4 (Web Text))

Max Activating Token Index: 284

Text #12

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.2187. Min Act: -0.0000

Data Index: 395157 (C4 (Web Text))

Max Activating Token Index: 712

Text #13

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3214. Min Act: -0.0000

Data Index: 240506 (C4 (Web Text))

Max Activating Token Index: 884

Text #14

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.4486. Min Act: -0.0000

Data Index: 979962 (C4 (Web Text))

Max Activating Token Index: 288

Text #15

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.2108. Min Act: -0.0000

Data Index: 670903 (C4 (Web Text))

Max Activating Token Index: 372

Text #16

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3172. Min Act: -0.0000

Data Index: 355548 (C4 (Web Text))

Max Activating Token Index: 107

Text #17

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.1648. Min Act: -0.0000

Data Index: 76799 (C4 (Web Text))

Max Activating Token Index: 161

Text #18

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.2694. Min Act: -0.0000

Data Index: 534546 (C4 (Web Text))

Max Activating Token Index: 1014

Text #19

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.3691. Min Act: -0.0000

Data Index: 1302453 (C4 (Web Text))

Max Activating Token Index: 957