Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1848 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0636. Min Act: -0.0005

Data Index: 471107 (C4 (Web Text))

Max Activating Token Index: 845

Text #1

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0628. Min Act: -0.0005

Data Index: 393129 (C4 (Web Text))

Max Activating Token Index: 682

Text #2

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0592. Min Act: -0.0004

Data Index: 296560 (C4 (Web Text))

Max Activating Token Index: 961

Text #3

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0640. Min Act: -0.0005

Data Index: 1245718 (C4 (Web Text))

Max Activating Token Index: 468

Text #4

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0603. Min Act: -0.0005

Data Index: 1161645 (C4 (Web Text))

Max Activating Token Index: 670

Text #5

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0607. Min Act: -0.0005

Data Index: 884393 (C4 (Web Text))

Max Activating Token Index: 952

Text #6

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0592. Min Act: -0.0005

Data Index: 791536 (C4 (Web Text))

Max Activating Token Index: 542

Text #7

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0643. Min Act: -0.0005

Data Index: 595283 (C4 (Web Text))

Max Activating Token Index: 684

Text #8

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0607. Min Act: -0.0004

Data Index: 750997 (C4 (Web Text))

Max Activating Token Index: 153

Text #9

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0565. Min Act: -0.0004

Data Index: 381792 (C4 (Web Text))

Max Activating Token Index: 1001

Text #10

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0568. Min Act: -0.0005

Data Index: 1083730 (C4 (Web Text))

Max Activating Token Index: 881

Text #11

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0601. Min Act: -0.0005

Data Index: 1160934 (C4 (Web Text))

Max Activating Token Index: 806

Text #12

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0574. Min Act: -0.0005

Data Index: 646015 (C4 (Web Text))

Max Activating Token Index: 75

Text #13

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0560. Min Act: -0.0004

Data Index: 830064 (C4 (Web Text))

Max Activating Token Index: 817

Text #14

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0548. Min Act: -0.0005

Data Index: 1270673 (C4 (Web Text))

Max Activating Token Index: 900

Text #15

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0526. Min Act: -0.0005

Data Index: 1219425 (C4 (Web Text))

Max Activating Token Index: 705

Text #16

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0551. Min Act: -0.0005

Data Index: 648023 (C4 (Web Text))

Max Activating Token Index: 212

Text #17

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0518. Min Act: -0.0004

Data Index: 1340895 (C4 (Web Text))

Max Activating Token Index: 200

Text #18

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0518. Min Act: -0.0005

Data Index: 786139 (C4 (Web Text))

Max Activating Token Index: 392

Text #19

Max Range: 0.0643. Min Range: -0.0643

Max Act: 0.0517. Min Act: -0.0005

Data Index: 76795 (C4 (Web Text))

Max Activating Token Index: 173